• <tr id="yyy80"></tr>
  • <sup id="yyy80"></sup>
  • <tfoot id="yyy80"><noscript id="yyy80"></noscript></tfoot>
  • 99热精品在线国产_美女午夜性视频免费_国产精品国产高清国产av_av欧美777_自拍偷自拍亚洲精品老妇_亚洲熟女精品中文字幕_www日本黄色视频网_国产精品野战在线观看 ?

    Deep Learning Based Attack Detection for Cyber-Physical System Cybersecurity: A Survey

    2022-01-26 00:35:22JunZhangLeiPanQingLongHanChaoChenShengWenandYangXiang
    IEEE/CAA Journal of Automatica Sinica 2022年3期

    Jun Zhang,,Lei Pan,,Qing-Long Han,,Chao Chen,,Sheng Wen,,and Yang Xiang,

    Abstract—With the booming of cyber attacks and cyber criminals against cyber-physical systems (CPSs),detecting these attacks remains challenging.It might be the worst of times,but it might be the best of times because of opportunities brought by machine learning (ML),in particular deep learning (DL).In general,DL delivers superior performance to ML because of its layered setting and its effective algorithm for extract useful information from training data.DL models are adopted quickly to cyber attacks against CPS systems.In this survey,a holistic view of recently proposed DL solutions is provided to cyber attack detection in the CPS context.A six-step DL driven methodology is provided to summarize and analyze the surveyed literature for applying DL methods to detect cyber attacks against CPS systems.The methodology includes CPS scenario analysis,cyber attack identification,ML problem formulation,DL model customization,data acquisition for training,and performance evaluation.The reviewed works indicate great potential to detect cyber attacks against CPS through DL modules.Moreover,excellent performance is achieved partly because of several highquality datasets that are readily available for public use.Furthermore,challenges,opportunities,and research trends are pointed out for future research.

    I.INTRODUCTION

    CYBER-physical systems (CPSs) suffer from cyber attacks when they are increasingly connected to the cyber space.According to [1] published in 2017,more than 30 surveys were published to cover the cybersecurity issue in the CPSs.Cyber attacks have become increasingly sophisticated and prevalent as automated attacking tools,and professional hacking groups have started to get involved.A successful cyber attack against a CPS may be disastrous,catastrophic,or even fatal [2]–[6].However,it is a challenge to defend against cyber attacks on CPSs.Many CPS systems lack cybersecurity mechanisms like message authentication,resulting in challenges to detect false data injection attacks.A lack of universal encryption,especially on the systems employing dated technologies,makes it challenging to defend against eavesdropping attacks.System states need to be referred to detect replay attacks.In addition,the use of dated technology in operation limits the choices of defenses to network traffic in most cases [7].

    Deep learning (DL) [8],[9] delivers superior performance to traditional machine learning (ML) solutions.Whenever there is adequate data,DL models almost deliver excellent results.However,DL models have been slowly applied to solve the CPS cybersecurity issue compared with other fields such as NLP,image processing,software vulnerability [10],[11],and many more [12]–[17].It is also observed that many DL models have been proposed in recent publications to detect CPS cyber attacks.A widely accepted view to explain the difficulty of detecting cyber attacks on CPSs was accredited to the degree of complexity when superposing cybersecurity over CPSs [2].

    There exist a few short-length survey papers on CPS cybersecurity [1],[2],[18],[19].Some papers investigated data-driven methods for detecting cyber attacks against CPS systems [18],[20].However,there is no detailed discussion on applying DL methods to detect CPS cyber attacks.A short survey was provided in [18] with a four-step framework to apply DL methods on CPS issues,including cybersecurity,adaptability,recoverability,and many more,without a specific focus on cybersecurity.Furthermore,most of the cited works in [18] were published between 2012 and 2016,but this survey includes most papers between 2017 and 2021.A survey of surveys was presented in [1] without relevance to DL models.A comprehensive survey on the cyber attacks against CPSs was presented in [20] without investigating the DL models.Various methods of detecting cyber attacks in the CPSs were summarized in [2] without using DL methods.A comprehensive list of CPS attacks and challenges were provided in [19] but overlooking ML,or DL approaches.A cybersecurity analysis framework was proposed in [21]without utilizing the rich sources of available data.A recently published survey in [22] presents cybersecurity control and state estimation from active and passive defence perspectives.

    Fig.1.The DL driven methodology for CPS cybersecurity considers the essential needs for training robust and usable DL models in the context of cyber attacks against the CPS systems.

    We aim to review current research works on the advances of DL driven solutions for detecting cyber attacks in the CPS domain.It provides an overview for readers to quickly understand and step into the field by following our six-step DL driven methodology.Our six-step methodology considers the complete cycle of DL application from broad scenarios to performance evaluation.This paper caters to researchers,practitioners,and students interested in building DL-based cybersecurity applications in CPSs.The key contributions of this survey are three-fold:

    1) We conduct an up-to-date review of detecting cyber attacks in CPSs using DL models and propose a six-step methodology to position and analyze the surveyed works.

    2) We provide an overview for the state-of-the-art solutions with preservation of technical details.

    3) Based on the methodology,we discuss the challenges and future research directions.

    The rest of this survey is organized as follows: Section II proposes a research methodology for deep learning driven CPS cybersecurity.Section III presents the reviews on stateof-the-art research.Section IV discusses the research challenges and future work.Finally,Section V concludes this survey.

    II.RESEARCH METHODOLOGY

    Our methodology represents a deep understanding of the surveyed papers.The process consists of six steps,including CPS scenario analysis,cyber attack identification,DL problem formulation,DL model construction,data acquisition,and performance evaluation.Fig.1 shows a process of detecting cyber attacks in the context of a CPS by using DL models.For example,a smart grid may suffer from erroneous controls derived by electric load forecasts [20],[23].Falsely injected messages containing maliciously crafted information need to be identified and eliminated before committing the prediction process.A stacked AutoEncoder(AE) proposed in [24] may serve as a reliable regressor to predict the energy load on the system.The chosen AutoEncoder was subsequently trained with sufficient simulation data.At last,the DL model delivered excellent prediction results with the mean absolute percentage error of 3.51% on annual predictions.

    A.Step I: CPS Scenario Analysis

    The normal operations of CPSs rely on several important factors,including dependability,real-time operation,fault tolerance,cybersecurity,and many more.We must consider these requirements holistically.Dependability consists of service availability and reliability to minimize the system downtime; real-time operation is a critical factor for maintaining the system operation when the inputs and environment rapidly change; fault tolerance requires that the critical components of the system have sufficient backups to prevent the system from shutting down; and cybersecurity requirements are becoming more and more prominent when many CPSs are connected to the cyber space to improve the quality of system control and the overall level of quality of service.According to Mitchellet al.[2],there are four primary categories of characteristics of CPS intrusion detection,including physical process monitoring,closed control loops,attack sophistication,and legacy technology.

    Physical process monitoring:Physical properties of a CPS should be constantly monitored to identify any anomalies of the system because many physical processes of the CPS follow the laws of physics.

    Closed control loops:CPS events are significantly more regular and predictable than user-triggered events because many CPS events are driven by the preset feedback-based controllers.

    Attack sophistication:Sophisticated cyber attacks are increasingly popular in the CPS context because the potentially huge payoff for a successful cyber attack may bring sensitive information,valuable intelligence for military or finance operations,and many more.

    Legacy technology:Legacy hardware commonly used in the CPSs cannot interact with software-defined control because of the existing mechanical and hydraulic control.

    Analyzing the characteristics of a CPS scenario will help craft an appropriate cybersecurity problem.The involvement of physical signals enriches the input variables and complicates the design of any security solutions for CPSs.Although the behaviors of simplified proof-of-concept systems are relatively regular and predictable,real-world systems often operate in a noisy environment with unprecedented cyber threats.

    B.Step II: Cyber Attack Identification

    Upon completion of identifying the CPS scenario,we need to define a set of appropriate cyber attacks associated with CPS characteristics.For example,we will have more confidence to detect the falsely injected network packets if physical processes of the CPS components are properly monitored; cyber attacks like replay attacks may be detected on a CPS with a closed control loop; unknown attacks and sophisticated attacks like web attacks need to be considered if there is any concern of attack sophistication; denial of service(DoS) attacks and replay attacks are more prevalent in the presence of legacy technology.

    Based on the surveyed articles,we identify many common cyber attacks.Some frequent cyber attacks against the industrial control network include false data injection attacks,DoS attacks,replay attacks,and alike; and some frequent cyber attacks against the software-based controllers with a centralized server include brute force attacks,botnets,web attacks,heartbleed attacks,infiltration attacks and many more.Effective and efficient detection of these cyber attacks can be leveraged by using DL models,so we will need to translate the cybersecurity problem to the ML domain.

    C.Step III: ML Problem Formulation

    After aligning the cyber attacks to the CPS characteristics,the research problem can be translated to the ML/DL domain.ML is defined in [25] as “A computer program is said to learn from experienceEwith respect to some class of tasksTand performance measureP,if its performance at tasks inT,as measured byP,improves with experienceE.” DL is referred to in [8] as solving a complex problem by using a hierarchy of more straightforward concepts without too much human intervention.The definition of ML is general,and we will implement an ML solution in multiple steps.In this step,we need to define the taskT,including classification,clustering,regression,etc.A classification task requires that the trained model allocates its output to a pre-defined set of “classes”which could be the specific cyber attack categories; a clustering task often requires that the trained model allocates its output to a few “clusters” which could indicate normal traffic or attack traffic; a regression task is also known as a prediction task which requires the trained model to predict some numerical values.For example,a classification problem was found in [26] to differentiate cyber attack types; a clustering problem was found in [27] to separate covert messages from the normal messages; and a regression problem was set in [24] to predict the electric load in a smart grid.The choice of the ML tasks will impact the construction of the DL models.

    D.Step IV: DL Model Customization

    The DL model is constructed by selecting an architecture suitable for the research problem and optimizing parameters.The choice of DL models should be made according to actual needs.For example,autoencoders are good at translating the input data so that they are suitable for learning the representations of the data often required in prediction or regression tasks [24]; convolution networks (CNNs) and other models are usually used in classification tasks [28].

    The configuration of the chosen DL model also depends on the available data.A DL model with a large number of neurons per layer will almost always require more data than a DL model with the same design but a few neurons per layer.Some trade-offs can also be made by stacking more hidden layers inside the DL model instead of expanding the layer size.The ways and insights of the customizing model can be explored based on a thorough understanding of DL algorithms and CPS cybersecurity data.Furthermore,we can achieve improvement at various levels by combining the choice of DL models with a specific research problem.

    E.Step V: Data Acquisition for Training

    Data acquisition is a critical step for training DL models.The quality and quantity of data determine the effectiveness of solving the research problem.Also,data can serve as the source for setting up ground truth and affect the prediction model’s performance.One of the simplest methods to collect data is through simulation.This method is often used to generate datasets for power grids such as IEEE 9-bus,14-bus,30-bus,and 118-bus systems in Matlab.The other method relies on several existing datasets harvested by other researchers.These datasets include the SWaT dataset1http://itrust.sutd.edu.sg/dataset/SWaT,the SCADA IDS dataset2https://sites.google.com/a/uah.edu/tommy-morris-uah/ics-data-sets,the CICIDS2017 dataset3https://sites.google.com/a/uah.edu/tommy-morris-uah/ics-data-sets,the UNSWNB15 dataset4https://www.unsw.adfa.edu.au/unsw-canberra-cyber/cybersecurity/ADFANB15-Datasets/,and the KDD99 Cup dataset5http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html.

    Different cyber attacks were included in the datasets:

    1) The SWaT dataset contained eleven days of network traffic collected from a scaled-down water treatment plant.And there were no attacks during the first seven days.It includes 36 types of cyber attacks that are most commonly seen in today’s CPS systems.

    2) The SCADA IDS dataset contained network traffic logs of a SCADA IDS system.It includes seven types of cyber attacks — injection random response packets,hide the real state of the controlled process,inject malicious state commands,inject malicious parameter commands,inject malicious function code commands,DoS attack,and recon attack.

    3) The CICIDS2017 dataset [29] contained network traffic logs collected from an industrial control system.It includes six types of cyber attacks — brute force attacks,botnet,DoS attack,web attack,heartbleed attack,and infiltration attack.

    4) The Bot-IoT dataset [30] contained network traffic logs collected from an IoT setup.It includes three types of cyber attacks — infiltration,DoS attack,and information theft.

    5) The power system dataset [31] contained the network traffic collected from a power grid.It includes three cyber attacks — data injection,remote command injection,and replay attacks.

    6) The UNSW-NB15 dataset [32] contained the network traffic information extracted form a 100 GB packet capture dump.It includes nine cyber attacks — DoS attacks,exploits,recon attacks,worms,fuzzers,web penetration attacks,backdoors,shellcode,and generic attacks against block ciphers.

    7) The KDD99 Cup dataset [33] contained the network traffic information presented at the ACM SIGKDD conference in 1999.It includes four cyber attacks — DoS attacks,unauthorized accesses,privilege escalation,and probing attacks.

    F.Step VI: Performance Evaluation

    The last step is used to determine whether the DL model meets our expected objectives through performance evaluation.The performance is usually measured according to various metrics.We divide the performance metrics into two categories according to the tasks: 1) For prediction or regression tasks,a number of error metrics are used to measure the performance,including mean absolute error(MAE),mean relative error (MRE),root of mean squared error (RMSE),and mean absolute percentage error (MAPE).2) For classification or clustering tasks,there are a few standard metrics,including accuracy,recall,precision,false positive rate (FPR),F1 score.And occasionally,graphical plots like receive operating characteristic (ROC) curves are used by plotting TPR asy-axis and FPR asx-axis to depict the trade-offs between benefits and costs.Finally,area under ROC curve (auROC) is used to indicate the cumulative strength of a particular ROC curve.

    In many cases,FPR poses challenges for the DL models because the false alarms almost always result in excessive costs associated with manual verification.And it is always challenging to detect the rare or even unknown attacks as proven in [34] so that most of the surveyed literature aimed to maximize the TPR while minimizing the FPR.On the other hand,the error rate can be tolerated more generously in the regression task than in the classification task.By leveraging comprehensive evaluation metrics,we can decide whether the outputs of a specific DL model are satisfactory.Whenever there are unsatisfactory results,the process should be repeated with proper adjustments.

    III.CPS CYBERSECURITY WITH DEEP NEURAL MODELS

    This Section surveys the relevant literature of detecting cyber attacks in the context of CPSs by following the research methodology described in Fig.1.In particular,the body of the literature is divided into two parts according to the DL architectures,which will be elaborated below.

    A.Representation Learning for Attack Detection

    An AutoEncoder-based (AE) model was proposed in [26] to preserve privacy information in the context of smart power networks.Data privacy violations are becoming more and more popular in smart power networks.It is challenging to defend against inference attacks,because the smart power networks represent the CPS characteristics of physical process monitoring,closed control loops,attack sophistication,and legacy technology.The research problem of defending against inference attacks was translated into a classification problem in the ML domain.A Variational AutoEncoder (VAE) was proposed to provide transformed features for the ultimate classification task and transform raw data into an encoded format for preventing inference attacks.A VAE is a feedforward model used for encoding an input into new data codes using a set of weighted parameters.The VAE consisted of one input layer,four hidden layers,and one output layer.The transformed data from the output layer were written to the database for publication.Two datasets were used to evaluate the VAE,i.e.,the power system dataset [31] and the UNSWNB15 dataset [32].The Power system dataset is a multi-class dataset involving 37 scenarios that include 8 natural events,28 intrusive events,and 1 no event; and the UNSW-NB15 dataset includes a combination of current normal and attack records.300,000 random samples of legitimate and attack observations were chosen from each dataset for assessing the performance of the proposed framework.Although the VAE was only employed as a part of the intrusion detection system,its strength was demonstrated while transforming complex data into a simple form.The VAE achieved 0.921 for accuracy and 0.005 for loss on the power system dataset,and 0.998 for accuracy and 0.0001 for loss on the UNSW-NB15 dataset.

    An AE-based solution was proposed in [35] to detect various cyber attacks in the context of industrial control networks.There exist many kinds of cyber attacks when control networks are connected to the internet.The research problem generally reflects the CPS characteristics of attack sophistication.And it was translated to a classification problem in the ML domain.Hence,a 7-layer AE consisted of an input layer,four hidden layers,and an output layer.The input layer had 41 units corresponding to the feature space’s dimension,and the output layer had five units corresponding to the five types of network traffic.In particular,the last hidden layer was a softmax layer to provide the stability of the model.The AE was trained using the NSL-KDD dataset [33].As an early study,the proposed AE suffered low performance in detecting small classes like probe attack and remote attack.The stacked AE achieved 0.978 for accuracy over the five categories.The model achieved an F1 score of 0.9683.

    An AE-based model was proposed in [24] to detect cyber attacks in the context of smart grids.One big challenge is a large number of control parameters.The smart power networks represent the CPS characteristics of physical process monitoring and legacy technology.The smart grid’s essential controller is based on state estimation,so the lower and upper bounds of each state variable need to be predicted as accurately as possible.Hence,this research problem was translated into a regression problem in the ML domain.A stacked AE (SAE) was proposed to process the smart grid data.The SAE consisted of an input layer,three vanilla AEs,and a logistic regressor as the output layer.The SAE was trained with simulated data representing IEEE 9-bus,14-bus,30-bus,and 118-bus systems.Overall,the SAE in this study achieved excellent results in predicting the electric load forecast.The mean absolute percentage error (MAPE) was used to evaluate the SAE’s accuracy.And the SAE achieved a MAPE of 3.51% on an annual prediction and outperformed the baseline models like SVM and BP.Despite the SAE model’s simplicity,the empirical studies showed its applicability and consistency in performing load forecasts.

    Another AE-based model was proposed in [36] to detect Phasor measurement unit data manipulation attacks (PDMAs)in smart grids.PDMAs are challenging to be detected because of the similarity between PDMAs and man-in-the-middle attacks with infiltration of communication networks.This problem represents the CPS characteristics of physical process monitoring and attack sophistication.The main idea was to detect anomalies based on the normal operation patterns from the data collected from the PMUs with PDMA-free measurements in a distributed manner.Hence,the research problem was translated to a regression problem in the ML domain.A deep AE (DAE) was constructed by stacking four RBM models.Training the deep AE required multiple stages by fine-tuning the intermediate RBMs.The input layer took 108 numerical features,and the output layer is a regressor.The dataset was collected from a simulated IEEE 9-bus system and had 250 000 records.The deep AE was trained by using 200 000 benign records,20 000 records were reserved as the validation dataset,and the testing dataset consisted of 30 000 samples with half from attack records.The studies showed that the deep AE outperformed the baseline models like OCSVM,C4.5,MLP,SVM,and kNN.The DAE achieved 0.941 for accuracy,0.996 for precision,0.886 for recall,and 0.9038 for F1 score.Despite the success of using the deep AE in this study,it is challenging to obtain benign data from real-world power networks,and new methods may need to be explored.

    An AE-based model was proposed in [37] to detect attacks against physical measurements in the context of smart grids.This problem represents the CPS characteristics of physical process monitoring and legacy technology.It is challenging to derive useful features for intrusion detection in a noisy environment in a real-world factory.Hence,the research problem was translated into a classification problem in the ML domain.A stacked denoising AE (SDAE) was proposed to learn the advanced features from the input data.And the learned features were fed to an ELM for classification.Simulated data from gas turbines were collected and used to train the model.The proposed model achieved excellent results with an FPR of 0.000006,which was significantly below the required FPR of 0.01.The SDAE was used as a part of the IDS model but demonstrated its strength in extracting useful features to represent physical measurements from a noisy environment.

    An AE-based model was proposed in [28] to detect cyber attacks in the context of the industrial control systems.This problem represents the CPS characteristics of physical process monitoring,attack sophistication,and legacy technology.Hence,the problem was translated into a classification problem in the ML domain.An AE was proposed to extract features for a 1D CNN classifier.The AE consisted of five layers,including an input layer,a corruption layer applying Gaussian noise to the input,a fully connected layer with an activation function,an encoder layer,a decoding layer as the output layer to generate the extracted feature.The SWaT dataset was used to train the model.In particular,the training time for the AE was less than half-second,which was significantly faster than the 1D CNN model.The AE achieved 0.890 for precision,0.827 for recall,and 0.844 for F1 score.In summary,the AE model was validated as a powerful and efficient method to extract useful features.

    An LSTM autoencoder architecture was proposed to detect cyber attacks in the context of the autonomous vehicles (AVs)[38].AVs are linked together by using communication technologies,and thus are vulnerable to network attacks,such as Denial of Service,replay ans spoofing attacks.Such attacks can be inferred from network traffic.Authors designed an LSTM autoencoder to detect these cyber attacks.Statistical features from network traffic were extract to represent the activities of AVs.The designed neural network architecture was consisted of two types layers,LSTM and fully connected layer.A number of LSTM layers were used to encode the representation of the transformed likelihood stream.Then the reconstructed output was produced by the fully connected layer.Two datasets,i.e.,Car Hacking dataset and UNSWNB15 were used to evaluate the proposed scheme.In particular,the proposed LSTM based autoencoder achieved 0.99 for precision,1.0 for recall,and 0.99 for F1 score in the Car Hacking dataset.While on UNSW-NB15 dataset,the proposed scheme achieved 0.1 for precision,0.97 for recall,and 0.98 for F1 score.In a word,this work can successfully detect multiple types of attack vectors.

    Remark 1:Research works employing AE-based architecture were summarized in Table I.Most of them focused on smart grids or power network systems.Due to the difficulties in smart grids’ control systems,most AE models were used to learn the useful features of an intrusion detection system or predict the electric load as an indicator of cyber attacks.Moreover,the AE models were relatively small in size,so that they could be trained in a short amount of time.

    B.Cyber Recognition with Deep Learning Methods

    1) Cybersecurity Pattern Recognition with Deep Neural Networks (DNNs):A DNN-based model was proposed in [39]to learn the communication patterns between electronics control units (ECUs) in the context of in-vehicular network security.The security of communication messages among ECUs is vital because a group of ECUs can control and monitor a vehicle’s status during a maneuver.It is challenging to ensure cybersecurity because most communications between ECUs are through the controller area network protocol,which has no support for authentication or integrity check.Specifically,fake packets injected into the open communication channel through the controller area network protocol pose severe cybersecurity risks.Detecting the fabricated or modified packets in the vehicular setup needs to meet the requirements of physical process monitoring and legacy technologies.This intrusion detection problem was translated into a binary classification problem in the ML domain.That is,statistical features were extracted from highdimensional CAN packet data through a dimension reduction process to represent the normal and attack packets.A 5-layered DNN model was constructed based on a standard DBN model by adding a binary classification layer as the final output layer.The DBN’s coefficient weights were determined through an unsupervised pre-training process,but the final DNN model was trained with a bottom-up supervised manner.During each simulation round,a total of 200 000 packets were generated by the Open Car Test-bed and Network Experiments(OCTANE) generator.A 70:30 split was made to divide training and testing sets.Many experiments were conducted by varying the layers of the DNN model from 5 to 11 to investigate the trade-offs between performance and efficiency.The empirical results demonstrated the effectiveness of the proposed DNN model while comparing it with ANN and SVM.The best performance was achieved as 0.978 foraccuracy,0.016 for false positive rate,and 0.028 for false negative rate.Given the detection ratio of over 99%,the proposed DNN model showed good potentials to detect fake packets on vehicular networks despite that the DNN models’efficiency with more than five layers needed to be improved to meet the real-time requirements.

    TABLE IRESEARCH WORKS EMPLOYING AUTOENCODERS (AES)

    Another DNN-based model was proposed in [40] to learn the network traffic patterns in the electric power grid context.The cybersecurity of an electric power grid largely depends on state estimation underpinning critical control processes for the grid.It is challenging to detect false data injection attacks against the state estimation because a skilled cyber attacker may disguise the injected data stealthily with the inside knowledge of system topology.Such successful attacks may blackout an entire region due to the falsely impacted state estimation because the injected data value is progressively added to the legitimate signal and the Gaussian noise values.Detecting the injected data fed to the state estimation model needs to meet the requirements of physical process monitoring and closed control loops in a power grid.Hence,this intrusion detection problem was translated into a binary classification problem with the objective function of simultaneously minimizing the number of false positives and false negatives.A series of measurement vectors were created for a specific time slot so that a compromised vector contains any injected component corresponding to a false data inject attack.Four variations of DNN models were constructed with different settings — 1 or 3 hidden layers,100 or 150 neurons per hidden layer,whether to use L1 regularization.The DNN models were trained with the standard stochastic gradient descent approach using the back-propagation method.The activation function was tanh.The most accurate DNN model was also the most complex among all the four DNN models.In terms of accuracy,the DNN model of 3 hidden layers with 150 neurons on each layer without L1 regularization outperformed generalized linear models,gradient boosting machines,a distributed random forest classifier,and the other three DNN models.The best performance was 0.9802 for precision,0.9895 for recall,0.9852 for F1 score,and a low false alarm rate of 0.1840.The proposed DNN model demonstrated the effectiveness of a simulated IEEE 14-bus power grid without testing realistic datasets generated by Real-time Digital Simulation (RTDS) and physical testbeds.

    A DNN-based model was proposed in [41],[42] to detect the anomalies in the context of secure water treatment(SWaT).The SWaT system represents many challenges of securing CPSs,including physical process monitoring,closed control loops,attack sophistication,and legacy technologies.A divide-and-conquer strategy was employed in this scenario by separating different sets of sensors and actuators into groups according to their functionalities.Thereafter,the high dimension and complexity of detecting anomalies were mitigated.Data points with a low probability were regarded as outliers because common processes in the system usually generated the normal data points.The proposed DNN model included an LSTM layer and 100 intermediate layers.The LSTM layer predicted the actuator’s position based on its historical positions.Each hidden layer was fully connected to an output layer with a bi-linear function.The cost function was defined by the cross-entropy of the real probability distribution and the predicted probability distribution.The dataset was the log entries collected from 51 sensors and actuators of a testbed for 11 days.According to [41],[42],the DNN detected 13 of the total 36 scenarios,and the DNN model achieved 0.98295 for precision,0.67847 for recall,and 0.80281 for F1 score.However,it took two weeks to train the DNN model and 8 hours to complete testing the data.Due to its inefficiency,DNN has limited use for real-world applications.

    A DNN-based model was proposed in [27] to detect covert message transmission in the context of a chemical process plant.The covert messages containing critical control information exfiltrated from the actuators can be used to detect anomaly operations on hardware devices.Because the covert channel was established without any modification to the system,conducting data analytics on the covert channel had to involve CPS characteristics,including physical process monitoring and closed control loops.It is challenging for simplistic solutions to detect the messages transmitted via a covert channel when analog emission from the physical instrument was used to disguise the existence of the messages.Hence,the problem of detecting covert messages was translated to a clustering problem.A 10-layers DNN received the inputs from the digitized audio samples and produced binary outputs to indicate whether there was a covert message transmission or not.The ten layers included two dropout layers,four linear layers,three ReLU activation functions,and a tanh activation function.The DNN model was trained using Adam optimizer to maintain the learning quality.The dataset consisted of 9 minutes of audio recordings recorded by a Hardware-In-The-Loop (HITL) simulator.Due to the nature of analog signals,the performance of the DNN model was measured by an accuracy of 0.95 on testing data in online operation.The successful application of the DNN model opened many opportunities to develop and deploy real-time monitoring mechanisms over critical actuators using audio sampling and covert channels.

    2) Cybersecurity Pattern Recognition with Convolutional Neural Networks (CNNs):A CNN-based model was proposed in [43] to detect cyber attacks in the context of an industrial water treatment plant.Because the SWaT dataset was used in this study,it inherited many CPS characteristics,including physical process monitoring,closed control loops,attack sophistication,and legacy technologies.An anomaly detection approach was used for improving the detection rate of the cyber attacks on the SWaT dataset.A 1D CNN model was constructed by stacking a set of 1D convolution layer,a ReLU function,and a 1D max-pooling layer before applying to flatten,dropout,and a final fully connected layer.Among all the 36 different attack scenarios,the CNN model successfully detected 31.The CNN model achieved 0.912 for precision,0.861 for recall,and 0.886 for F1 score.When using 8 layers,the 1D CNN model reached 0.967 as the AUC value with only 15 535 parameters.This model’s training time and testing time for one epoch were 88 seconds and 47 seconds,respectively.Based on the positive results of this empirical study,the proposed CNN model achieved excellent results in detecting cyber attacks on the SWaT dataset and demonstrated good potentials in real-world deployment.

    A CNN-based model was proposed in [44] to detect message injection attacks in the vehicular networks.Due to the lack of security protections to the controller area network(CAN),network packets with malicious contents can be easily injected to the CAN bus resulting in gaining control of the vehicle.The detection of malicious packets on a physical vehicle needs to be conducted according to the CPS requirements.Two Raspberry Pi devices acting as a listener and an attacker were connected to an operational passenger vehicle via the OBD-II port to validate the model’s effectiveness.The attacker had four attacking modes — DoS attack,fuzzy attack,drive gear spoofing attack,and engine RPM gauge spoofing attack.It is challenging to effectively and efficiently detect all of the attacks in real-time.A deep CNN model named Inception-ResNet was employed with two blocks of convolution layers,pooling layers,a fully connected layer,and a softmax layer before generating the final output.The activation function was set to the ReLU function.The whole dataset was split into four sub-datasets according to the attack scenarios.The CNN model outperformed other classifiers,including LSTM,ANN,SVM,kNN,NB,and Decision Trees.Each attack lasted for 3 to 5 seconds during the four recording sessions of forty minutes each.Thus,each dataset contained 300 injection attack instances.The number of messages in the four attacking scenarios are 3 078 250 (DoS normal),587 521 (DoS attack),3 347 013 (fuzzy normal),491 847 (fuzzy attack),2 766 522 (gear normal),597 252 (gear spoofing),2 290 185 (RPM normal),and 654 897 (RPM spoofing).False negative rate (FNR) and error rate (ER) were used to measure the CNN model’s performance.In terms of FNR,the CNN model achieved 0.06,0.07,0.10,and 0.24 for gear spoofing,RPM spoofing,DoS attack,and fuzzy attack,respectively; in terms of ER,the CNN model achieved 0.03,0.04,0.05,and 0.18 for DoS attack,RPM spoofing,gear spoofing,and fuzzy attack,respectively.The empirical studies showed that the CNN model was a promising technique for detecting false message injection attacks to vehicular networks.

    A federated deep learning scheme (DeepFed) in [45],to detect cyber threats targeting industrial CPSs.Li et al.firstly designed a CNN-GRU based intrusion detection model.Second,they took account into the federated learning scenario,and built a framework to allow multiple industrial CPSs to build an intrusion detection model together.Lastly,they applied a secure communication protocol based on Paillier cryptosystem to preserve the privacy of model parameters.

    Industrial CPSs are not only targeted by traditional cyber threats,such as DoS attacks,but also by some cyber threats which are customized to industrial systems,such as response injection attacks.Furthermore,in the federation based CPS cyber threats detection framework,eavesdropping attacks on data sources and model parameters are emerging.Thus,the authors proposed DeepFed,which was based on CNN-GRU to detect such threats.The model architecture was consisted of a CNN module,a GRU module,an MLP module,and a softmax layer.To evaluate the performance of DeepFed,they ran experiments on a real-world dataset collected from a gas pipelining system,with 80% splited for training and the rest 20% for testing.The performance varied when different number of local agents (K) were in consideration.If K = 3,DeepFed could achieve an accuracy,precision,recall,F-score of 99.20%,98.86%,97.34%,and 98.08%.Overall,all metrics could reach over 97%.The experiments demonstrated the effectiveness of DeepFed to detect different types of cyber threats to industrial CPSs.

    3) Cybersecurity Pattern Recognition with Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM)Models:An LSTM-based model was proposed in [46] to identify anomaly sensor behaviors in the water treatment plant context.Detecting the cyber attacks in a real plant is challenging because of the complex setups of the sensors and actuators representing many CPS characteristics,including physical process monitoring,closed control loops,attack sophistication,and legacy technologies.Specifically,the anomaly behaviors and the sensors under attack needed to be identified at the same time.The LSTM model using a cumulative sum was proposed to learn the sensors’ behaviors to achieve a low false positive rate.The LSTM model consisted of three LSTM stacks with 100 hidden units each.The cumulative sum of the sequence predictions was introduced to reduce false positives because of their capabilities of indicating small deviations over time.And its loss function was a mean-square loss function.Ten attack scenarios of the SWaT dataset was used for training the LSTM model.The attack scenarios were made up of six single-stage-single-point attacks,two single-stage-multi-point attacks,one multi-stage-single-point attack,and one multistage-multi-point attack.It took about 24 hours to train the LSTM model,which is a big drawback.Nine out of ten attack scenarios were detected without listing detailed results for specific performance metrics.This work was an early study of applying DL models on the SWaT dataset with only limited success.Nevertheless,it showed good potentials for applying DL models for detecting cyber attacks against CPSs.

    An LSTM-based model was proposed in [47] to learn the traffic pattern generated by unknown or zero-day attacks in the context of a control system for a gas pipeline.Detecting unknown or zero-day attacks is challenging because most intrusion detectors on industrial control systems are manually configured for specific protocols and systems.A hybrid strategy of combining packet contents and temporal dependencies and a fast signature-based Bloom filter were used to triage anomaly traffic based on the network packets’contents.A subsequent and slow LSTM model was dedicated to capturing the unknown or zero-day attacks based on the time-series information.The proposed LSTM model consisted of two LSTM layers and a softmax layer.Each LSTM layer had 256 fully connected neuron cells,and the output softmax layer had 613 bits to match the length of signatures used by the Bloom filter.A dataset of network logs collected from the gas pipeline was used to train the model,where there were seven types of cyber attacks with a total of 214 580 normal network packets and 60 048 attack packets.Combining a fast Bloom filter and a slow but powerful LSTM model demonstrated its success in detecting cyber attacks in a small SCADA system.Regarding the detection ratio (recall),the hybrid model outperformed the other six classifiers for six out of seven attacks,including Bloom filter,Bayesian network,SVDD,isolation forest,PCA-SVD,and Gaussian mixture model.The proposed model also achieved 0.92 for accuracy,0.94 for precision,0.78 for recall,and 0.85 for F1 score.This study showed the promising application of the LSTM model to detect time-series anomalies supplemented by an efficient and lean model like Bloom filter.

    An RNN-based model was proposed in [48] to detect various cyber attacks in the context of smart grids.There are various cyber attacks against smart grids,such as DoS attacks,data infiltration,and so on.Detecting all the attacks on a large scale smart grid network is challenging because of the CPS characteristics in terms of attack sophistication and legacy technology.Hence,a vanilla RNN model was trained by using the truncated backpropagation through time (BPTT)algorithm.Three datasets were fed to the RNN model to cover a wide range of cyber attacks.In particular,the CICIDS2017 dataset [29] included brute force attacks,botnets,DoS attacks,web attacks,heartbleed attacks,and infiltration attacks; the Bot-IoT dataset [30] consisted of infiltration,DoS attack,and information theft; and the power system dataset [31] included data injection,remote command injection,were used to replay attacks.The CICIDS2017 dataset contains 2 830 743 records,the Bot-IoT dataset 73 360 900 records,and the power system dataset 78 404 records.Experiments were conducted on the three datasets separately,and the results showed that the RNN model outperformed the benchmark classifiers like SVM,random forest,and NB in terms of mean false positive rate —0.00986 for the CICIDS2017 dataset,0.01281 for the Bot-IoT dataset,and 0.03986 for the power system dataset.In terms of accuracy,the RNN model achieved 0.98941 for the CICIDS2017 dataset,0.99912 for the Bot-IoT dataset,and 0.96882 for the power system dataset,respectively.Although the proposed vanilla RNN performed well across the dataset,its integration with the blockchain component of the DeepCoin system for detecting fraudulent transactions remains unclear but may inspire further research works to use blockchain technologies together with DL models.

    An LSTM-based model was proposed in [49] to detect anomalous automatic dependent surveillance-broadcast (ADSB) messages transmitted between airplanes and control towers.A typical ADS-B message includes an aircraft’s flight information,such as flight number,speed,GPS coordinates,altitude,and many more.Due to historical design and implementation,the ADS-B system is an open messaging system without authentication or encryption.Therefore,it is challenging to defend ADS-B systems against malicious message injection attacks in the aviation industry because the ADS-B messages are transmitted very frequently at an average rate of 4.2 messages per second.Detecting the spoofed ADS-B messages represents the CPS characteristics of legacy technology.Hence,the problem was translated into an unsupervised anomaly detection problem in the ML domain.An LSTM model was proposed to capture the message sequences to detect anomalies according to estimated credibility scores.The LSTM model consisted of two vanilla RNN models stacked as an encoder-decoder paradigm.The LSTM was trained to reconstruct sequences of benign messages with minimal errors,where the reconstruction error score was calculated by using the cosine similarity.A largescale flight tracking dataset namedFlightradar24was used to train the LSTM model using flight data collected from 13 international airports.The LSTM achieved 1.00 as recall,0.03 as FPR,and 0.99 as TPR.The LSTM outperformed the baseline anomaly detection methods,including HMM-GMM,one-class SVM (OCSVM),local outlier factor (LOF),isolation forest (IF),and DBSTREAM.In particular,the spoof messages through RND and ROUTE attacks were detected almost instantaneously,but the SHIFT Down and SHIFT Up attacks were detected with some delay.The LSTM model achieved excellent results in detecting the spoofing messages.

    An LSTM-based model was proposed in [50] to predict multimedia data requests transmitted in the cyber-physical industrial networks.Multimedia data as the carrier for many cyber attacks are manifold,including image files,audio files,and many more.The prediction of multimedia data through a cache resource allocation system became a new proposal to defend the underlying network,representing the CPS characteristics of attack sophistication.Hence,this research problem was translated as a regression problem in the ML domain.A vanilla LSTM model was proposed to capture the spatio-temporal relations of the multimedia contents.Specifically,the LSTM consisted of 3 layers where the input layer’s size was equal to the number of multimedia types,and the hidden layer’s size was half of the input size.A two-yearlong network traffic logs collected from a factory-intensive area in Tianjin,China,were considered the normal traffic of the dataset; all cyber attacks were manually added to the dataset.And various ratios of training data and testing data were used to train the LSTM model.The empirical studies showed that the LSTM accurately predicted the multimedia contents.The LSTM outperformed three baseline models:SVM,ANN,and RNN.The performance was measured in MAE,MRE,and RMSE,where the LSTM achieved 3.9857 for MAE,0.0553 for MRE,and 4.6166 for RMSE.Despite the LSTM’s excellent performance,predicting multimedia contents in the industrial network remains open.

    4) Cybersecurity Pattern Recognition with Deep Belief Network (DBN):A DBN-based model was proposed in [51] to detect false data injection attacks in the context of energy internet.It is challenging to detect stealthy cyber attacks due to many control signals and meter reading data on the energy internet.And this represents the CPS characteristics of physical process monitoring,closed control loops,and legacy technology.In this study,the detection problem was formulated as a dual bi-level programming problem with the upper and lower bounds because the variations of the electric loads can be predicted.And the prediction problem is translated as a regression problem in the ML domain.A DBN model was trained to forecast electric load.The DBN was made of three stacked RBM models forming six layers — an input layer,four hidden layers,and a logistic regression layer as the output layer.The DBN was trained with the data collected from simulated IEEE 14- and 118-bus systems.An overall error rate was used to measure the DBN model’s performance,and the DBN achieved a 2.73% error rate that was almost 3% lower than the benchmark model SVM.In this study,the DBN used only as a forecasting component of the intrusion detection model is a good example of trading off between the DL model and other programming solutions.

    A DBN-based model was proposed in [52] to detect false data injection attacks in electric power networks.The detection of false data injection attacks represents the CPS characteristics of physical process monitoring,closed control loops,and legacy technology.The research problem was translated into a classification problem in the ML domain.The detection accuracy was investigated through the features automatically generated by DL models.Specifically,a DBN model was proposed as a baseline approach to compare with an RNN model and a graph neural network (GNN) model.The DBN was created by using the default settings of Tensorflow package without optimization.The data was collected from two simulated systems,IEEE 30-bus and 118-bus.The DBN achieved 0.9939 for precision and 0.98231 for recall on the IEEE 30-bus,which was almost identical to the GNN model and slightly higher than the RNN model.However,on the IEEE 118-bus,the DBN model’s performance was consistently better than the RNN model and comparable to the graph neuron network model.Overall,the DBN model shows consistency,and the GNN model shows some promising future.

    Remark 2:Table II lists thirteen different DL models for detecting various cyber attacks across various CPS scenarios.Among the models,four were DNN-based,two CNN-based,five RNN-based,and two DBN-based.It took longer time and more computational resources to train some models than others because of the nature of the architectures and parameter settings.In general,CNNs,DNNs,and DBNs were faster to train than RNNs and LSTMs.But the slower models like LSTMs or even an embedded LSTM layer had their merits in forecasting electric power load soon.On the contrary,DNNs,CNNs,and DBNs provided more powerful capabilities to detect anomalies.Last but not least,DNNs earned their popularity partly because of their simple setup and various hyperparameter fine-tuning techniques.

    IV.CHALLENGES AND FUTURE OPPORTUNITIES

    Six potential areas are depicted in Fig.2 where challenges and new research directions may arise.These six areas correspond to the six steps of our research methodology,as shown in Fig.1.Our research methodology helps provide an overview of the research literature and extract important elements for comparative analysis.The analysis results underpin research challenges and opportunities in the near future.

    A.New CPS Cybersecurity Scenarios

    The papers studied the communication networks in CPS scenarios.The majority of the surveyed publications investigated the CPS scenarios in water treatment plants or smart grids,which accounted for thirteen out of twenty surveyed papers.Among the remaining six papers,two studied vehicle networks,one on generic industrial control networks,one on chemical process plant,one on aviation communication networks,and one on gas pipeline controllers.The imbalanced topics suggest that CPS scenarios were underrepresented.

    Applying DL to the 21st-century manufacturing industry is an emerging topic.DL has been used to detect flaws anddefects during the manufacturing process of complicated items such as semiconductor [53],and 3D printing [54].They were excluded because no cybersecurity issues were covered.A hybrid model was proposed in [53] to detect wafer defect patterns during the semiconductor fabrication process,and an AE-based model was proposed in [54] to detect defects in the 3D printed objects.These two works primarily considered the causes of defects as a production issue or a modeling issue without the presence of cyber attacks.However,cyber attacks and threats may exist in the manufacturer’s network or in the cloud server where the design models are stored.Moreover,blockchain technologies have been studied in two of the surveyed papers [26],[48] in smart grids for providing additional cybersecurity and privacy guarantee.We anticipate that blockchain will be studied much more widely together with CPS.And the diversity and development of CPS scenarios will require intensive studies on cybersecurity.

    TABLE IICYBERSECURITY PATTERN RECOGNITION WITH DEEP LEARNING

    B.Identification of New Cyber Threats

    Almost all the surveyed papers studied false data injection attacks.The detection of stealthy false data injection attacks is challenging because of the large amount of noise produced in the CPS and the lack of cybersecurity mechanisms to authenticate devices and messages transmitted across the network.There are a few types of false injection attacks,depending on the attacker’s information and goals.For example,no advanced information is required for launching DoS attacks; only recorded packets are required for replay attacks; scanning tools are required for probing attacks;automated tools are required for fuzzy attacks.The effective and efficient analysis of the network traffic is crucial for defending against these attacks [55]–[57].

    Furthermore,CPS cybersecurity is much broader than cyber attacks against CPS systems.Detecting the cyber attacks that originated in the cyber space and penetrated the physical domain remains a great challenge.For example,the Stuxnet attack [58] utilized many cyber attack elements like system vulnerabilities,network sniffing,and many more,to exploit the kinetic system.Such attacks may cause significant damage and loss to the physical infrastructure.There are proposals to mitigate these sophisticated attacks by monitoring the CPS system’s cyber component and the physics component.However,it remains an open problem to correlate the interdependent monitoring mechanism in a distributed and realtime manner.

    Fig.2.Research directions for DL driven CPS cybersecurity.

    Considering the characteristics of CPS cybersecurity,it needs to explore three topics — advanced persistent threats,insider attacks,and cyber incident forecasts.Advanced persistent threats (APTs) quickly appear due to the availability of attack tools and techniques and are difficult to defend against because of their frequently updated signatures and evolving behaviors based on reconnaissance results [59].Insiders are very challenging to defend against because their specialized knowledge can be utilized to circumvent security checkpoints deployed on the perimeters of the critical infrastructure [60].Despite its potential high false positives,cyber incident forecasting is valuable to prepare early for fortifying strong defense and improve the overall cyber resilience [61].Due to available big data,many aspects of these three topics can be significantly advanced and improved by adopting ML and DL models.In particular,new insights and knowledge can be obtained via visualization,and deep analysis.We expect that emerging cyber attacks will precede the defense mechanisms,but the risk can be mitigated through the data-driven approach.

    C.Adopting New ML/DL Paradigms

    All the surveyed papers followed traditional ML paradigms,including supervised and unsupervised learning.Apart from four papers that examined regression problems and three papers on clustering problems,the remaining papers studied classification problems.The dominating use of the supervised learning paradigm reflected the importance of the well-labeled data.In particular,network packets were labeled as normal or attack traffic,and the attack types often were differentiated.Such reliance on labeled data restricted the wide adoption of ML or DL methods.We advocate for researchers and practitioners to try new ML/DL paradigms.These new paradigms include reinforcement learning and self-supervised learning,together with improving the model’s explainability.Instead of relying on learning the records,reinforcement learning focuses on experience [62].It is very suitable in isolated environments where many CPSs are currently deployed.For example,a reinforcement learning model was constructed in [63] to predict the normal driving behaviors.Deep reinforcement learning was applied to build human-level intelligence [64].Deep reinforcement learning may have huge potentials in fighting against cyber attacks on CPSs.Selfsupervised learning is a relatively new paradigm to generate data labels automatically.Self-supervised learning is a special kind of supervised learning without the need for the manual labeling process.In theory,self-supervised learning is ideal for changing environments where zero-day attacks and unknown attacks exist because the relationship between input data is deeply analyzed.A popular and successful model is BERT [65] for deriving deep bidirectional representations from unlabeled data.Because of BERT’s huge success in NLP and many other domains,we anticipate that self-supervised learning will prosper in the CPS domain because DL models generally suffer from poor explainability [66].We know how well the DL model works but do not know why the model works.Explainability is important for many CPS problems when we need to improve the system.Fortunately,tools like LEMNA [67] are available to explain DL-based cybersecurity applications.LEMNA can identify the critical features for a trained DL model like RNN and LSTM.We are optimistic to predict that the DL models will be more and more explainable when new tools and techniques are invented and used.

    D.Defending the Trained DL Models

    No surveyed works are considered to defend the trained DL models against various attacks.However,we would like to highlight the importance of defending the trained DL models because of the computational expenses to train the DL models,the important roles of the trained models,and potential dangers if the DL models are compromised.Due to the hunger for training samples,the DL models are sometimes trained with data from untrustworthy sources.Thus,adversarial attacks are prevalent because of linear behavior in highdimensional space [68].For example,Android HIV was proposed in [69] to automatically generate adversarial Android malware that the existing detectors failed to detect.

    Adversarial attacks are categorized into two types,including evasion attacks and poisoning attacks [70].Evasion attacks aim to evade a trained classifier by altering input data,and poisoning attacks aim to affect the trained classifier by injecting poisoning samples to the training dataset.These attacks differ by their time of occurrence.Poisoning attacks usually occur during the training phase,but evasion attacks occur during the test phase.Much research efforts have been devoted to fighting against data poisoning attacks,including robust optimization,adding some adversarial samples to training data,blind-spot removal,examining decision boundaries,and many more.There are many strategies to counter poisoning attacks,such as outlier removal,data sensitization,frequent model retraining,classifier ensembles,randomizing the classifier’s output,and many more.Last but not least,model stealing attacks were proven in [71] as a big threat to the trained model.The attackers can obtain sufficient information to mimic an ML or DL model by launching many queries and aggregating the results.The extracted information can be used to build a mimicking model for the attacker to find possible evasion attacks.We strongly advocate that cyber defense should be conducted as soon as possible due to the unawareness of adversarial attacks in the CPS scenarios.

    E.Enriching CPS Cybersecurity Datasets

    Among the surveyed papers,datasets collected in the field dominated the simulation with a 14:6 ratio.Simulated data were investigated in the two CPS scenarios — smart grids and vehicular networks.In smart grids,Matlab was the only choice for simulating electric load in five papers; and the OCTANE simulator was used in one paper on vehicular networks.However,there is a significant risk of solely relying on proprietary products like Matlab because the availability of such products may be discontinued unprecedently.On the other hand,field data is independent of the simulation platform and offers researchers good flexibility.Among the papers using field data,five papers chose the SWaT dataset,two papers the CICIDS2017 dataset,and the rest seven papers different datasets.The SWaT dataset dominated the field data category for a few reasons: 1) The network traffic data were continuously collected for 11 days from the control networks and from the sensors of a physical testbed,2) the traffic with and without attack were chronologically separated for easy use,and 3) there were 36 attack scenarios against different components of the testbed.To our surprise,the NSL-KDD dataset [33],also known as KDD99-cup,was studied in one surveyed paper despite its age of 20+ years.Using such a dated dataset may cause people to draw biased conclusions because many cyber attacks were not included in the NSLKDD dataset.Therefore,we recommend the researchers to use datasets like the UNSW-NB15 dataset [32] where recent cyber attacks were present.

    Moreover,new datasets will always be valuable and appreciated.Ideally,the new datasets are open-sourced field data collected from physical testbeds.Several CPS testbeds are proposed to facilitate detecting cyber attacks,such as[72]–[74].The recent trend of increasing interest in building CPS testbeds may benefit researchers to collect high-quality attack and defense data.The new datasets should be large enough to exploit DL models’ power,and both new and old cyber attacks should be included because cyber attacks evolve quickly.If labeling data is challenging,then chronologically separating the attacks from the normal traffic is a feasible idea.Artificially blending the data entries representing attacks into a set of normal traffic records should be avoided because the simple data augmentation method does not consider feasibility,attacking sequences,and possible changes of correlations.To benefit the advancement of research and knowledge,we strongly encourage more and more highquality datasets to be made available to the community.

    F.Improving the Model Evaluation

    Standard performance metrics were used in most of the surveyed papers.False positives were investigated,along with accuracy and error rates.It is proven in [34] that it is significantly more difficult to detect the rarely occurred attacks than the common ones derived by the Bayesian laws.In real-world CPSs,cyber attacks may rarely occur,so the DL models trained in the lab settings may be invalid.Since most papers did not investigate the impact of the imbalanced data between normal and attack traffic,the empirical results may be substantially biased or inflated.Cross comparisons in [75]showed that the precision-recall curve (PRC) and the area under precision-recall curve (auPRC) were more resilient to imbalanced data than ROC and auROC.Therefore,new studies should consider reporting PRC and auPRC.

    Furthermore,time decay should be considered in future studies because each trained ML or DL model’s performance will inevitably degrade over time.When the cyber attacks rapidly evolve,the models trained with old data will struggle with detecting new attacks.A time decay metric was proposed in [75] to evaluate a trained model’s performance loss.By studying the time decay,we will be able to decide when the model needs to be retrained.We strongly hope to see future work similar to [75] in the context of CPS and cyber attacks.Once the in-depth knowledge is developed and gained,we may expect to mitigate the risk of CPS cyber attacks.

    V.CONCLUSION

    This survey provides a current view of detecting cyber attacks in the CPSs.A six-step DL driven methodology is proposed to summarize and analyze the twenty recently published papers in this survey.Specifically,a panoramic view is obtained through inspecting the CPS scenarios,identifying cybersecurity problems,translating the research problem to the ML/DL domain,constructing the DL model,preparing datasets,and finally evaluating the model.Cyber attacks persist as an ongoing and prominent threat to the security and safety of the CPSs.The reviewed works show great potential to exploit CPS cyber data through DL models because of their promising performances.The excellent performance is achieved partly because of several high-quality datasets that are readily available for public use.In addition to following the success of current research,we also identified promising research topics,including integration with blockchain,detection of advanced persistent threats,adopting new ML and DL paradigms,prevention of adversarial and model extraction attacks,enriching datasets,and applications of additional performance metrics.We are optimistic and confident that the research in this field will flourish.

    九九爱精品视频在线观看| 亚洲国产欧美网| 男女啪啪激烈高潮av片| 2022亚洲国产成人精品| 精品国产一区二区久久| 激情五月婷婷亚洲| 国产一区亚洲一区在线观看| 国产精品久久久久久av不卡| 99久久精品国产国产毛片| 国产无遮挡羞羞视频在线观看| 亚洲精品中文字幕在线视频| 久久毛片免费看一区二区三区| 精品国产一区二区三区久久久樱花| 久久精品国产亚洲av高清一级| 欧美人与性动交α欧美精品济南到 | 国产精品av久久久久免费| 国产精品偷伦视频观看了| 国产成人精品婷婷| 不卡av一区二区三区| 午夜精品国产一区二区电影| 国产在线视频一区二区| 国产精品一国产av| 欧美日韩视频高清一区二区三区二| 2021少妇久久久久久久久久久| 在线亚洲精品国产二区图片欧美| 99国产精品免费福利视频| 亚洲av欧美aⅴ国产| 亚洲欧美成人综合另类久久久| 国产高清国产精品国产三级| 国产片特级美女逼逼视频| 熟妇人妻不卡中文字幕| 热99久久久久精品小说推荐| 国产精品 国内视频| 国产97色在线日韩免费| 高清不卡的av网站| 免费看av在线观看网站| 两个人免费观看高清视频| 香蕉丝袜av| 久久精品久久久久久噜噜老黄| 久久午夜综合久久蜜桃| 欧美成人午夜精品| 18+在线观看网站| 大话2 男鬼变身卡| 大码成人一级视频| 热99国产精品久久久久久7| 女的被弄到高潮叫床怎么办| 日韩,欧美,国产一区二区三区| 亚洲av中文av极速乱| 成人国语在线视频| 欧美日韩国产mv在线观看视频| 亚洲国产欧美日韩在线播放| 欧美激情 高清一区二区三区| kizo精华| 老鸭窝网址在线观看| 久久免费观看电影| 一级片'在线观看视频| av电影中文网址| 久久久久久人人人人人| 女性生殖器流出的白浆| 久久精品久久久久久噜噜老黄| 丰满迷人的少妇在线观看| 欧美精品亚洲一区二区| 另类亚洲欧美激情| 一区二区三区精品91| 一级毛片电影观看| 人妻系列 视频| 少妇被粗大的猛进出69影院| freevideosex欧美| 成人漫画全彩无遮挡| 国产精品99久久99久久久不卡 | 精品国产露脸久久av麻豆| 久久久精品国产亚洲av高清涩受| 久久国产亚洲av麻豆专区| 美女视频免费永久观看网站| 国产在线免费精品| 91aial.com中文字幕在线观看| 久久久久人妻精品一区果冻| 免费看av在线观看网站| 深夜精品福利| 日韩伦理黄色片| 免费观看性生交大片5| 老汉色∧v一级毛片| 国产毛片在线视频| 欧美日韩精品成人综合77777| 亚洲av电影在线观看一区二区三区| 97精品久久久久久久久久精品| 日韩一区二区视频免费看| 亚洲一码二码三码区别大吗| www.熟女人妻精品国产| 一区二区三区激情视频| 亚洲国产日韩一区二区| 少妇人妻 视频| 91精品伊人久久大香线蕉| 女人被躁到高潮嗷嗷叫费观| 波野结衣二区三区在线| 建设人人有责人人尽责人人享有的| 久久精品国产鲁丝片午夜精品| 精品人妻熟女毛片av久久网站| 成人手机av| 久久久久人妻精品一区果冻| 黄色 视频免费看| 亚洲成av片中文字幕在线观看 | 国产片特级美女逼逼视频| 午夜影院在线不卡| 一级片免费观看大全| 国产精品三级大全| 国产乱来视频区| 国产成人午夜福利电影在线观看| 中文欧美无线码| 亚洲情色 制服丝袜| 天堂俺去俺来也www色官网| 国产精品国产av在线观看| 日韩一本色道免费dvd| 欧美日韩亚洲国产一区二区在线观看 | 亚洲av欧美aⅴ国产| 免费久久久久久久精品成人欧美视频| 亚洲欧美日韩另类电影网站| 亚洲精品美女久久久久99蜜臀 | 男女边摸边吃奶| 亚洲内射少妇av| 免费日韩欧美在线观看| 国产午夜精品一二区理论片| 午夜av观看不卡| 精品少妇久久久久久888优播| 免费看av在线观看网站| 欧美最新免费一区二区三区| 可以免费在线观看a视频的电影网站 | 久久精品久久久久久久性| 国产不卡av网站在线观看| 久久免费观看电影| 精品少妇久久久久久888优播| 国产精品久久久久久久久免| 久久久久久久久久人人人人人人| 久久久久国产一级毛片高清牌| 色婷婷久久久亚洲欧美| 国产不卡av网站在线观看| 亚洲人成电影观看| 中文字幕av电影在线播放| 狠狠精品人妻久久久久久综合| 女性生殖器流出的白浆| 精品午夜福利在线看| 亚洲精品日本国产第一区| 国产精品香港三级国产av潘金莲 | 日本-黄色视频高清免费观看| 亚洲精品乱久久久久久| 侵犯人妻中文字幕一二三四区| 熟女av电影| 99香蕉大伊视频| 在线观看免费高清a一片| 高清在线视频一区二区三区| 日韩人妻精品一区2区三区| 国产日韩一区二区三区精品不卡| 韩国精品一区二区三区| 高清黄色对白视频在线免费看| 一级毛片电影观看| 亚洲精品国产av蜜桃| 久久久久久久久久人人人人人人| 欧美另类一区| 熟妇人妻不卡中文字幕| 电影成人av| 黑人欧美特级aaaaaa片| 在线观看人妻少妇| 免费人妻精品一区二区三区视频| 丰满少妇做爰视频| 蜜桃在线观看..| 波多野结衣一区麻豆| 99久久综合免费| 国产精品久久久av美女十八| 欧美日韩视频精品一区| 热99国产精品久久久久久7| 国产av一区二区精品久久| 久久精品亚洲av国产电影网| 国语对白做爰xxxⅹ性视频网站| 日韩视频在线欧美| 观看av在线不卡| av国产久精品久网站免费入址| 久久久久久久亚洲中文字幕| 香蕉国产在线看| 好男人视频免费观看在线| 亚洲三级黄色毛片| 日韩中字成人| 在线 av 中文字幕| www.av在线官网国产| 国产成人91sexporn| 成年人午夜在线观看视频| kizo精华| 一本色道久久久久久精品综合| 欧美日韩av久久| 欧美人与善性xxx| 久久这里只有精品19| 在现免费观看毛片| 精品人妻熟女毛片av久久网站| 亚洲精品乱久久久久久| 亚洲精品国产av蜜桃| 亚洲精品aⅴ在线观看| 午夜免费鲁丝| 久久久a久久爽久久v久久| 超色免费av| 9191精品国产免费久久| 人妻一区二区av| 成人国产av品久久久| 精品亚洲成a人片在线观看| 亚洲精品视频女| 亚洲国产欧美网| 午夜免费男女啪啪视频观看| 少妇人妻 视频| 国产成人aa在线观看| 亚洲中文av在线| 又粗又硬又长又爽又黄的视频| 久热久热在线精品观看| 制服丝袜香蕉在线| av国产精品久久久久影院| 欧美在线黄色| 亚洲综合色惰| 国产精品国产三级国产专区5o| 国产高清不卡午夜福利| 最近中文字幕高清免费大全6| 久久久久久久亚洲中文字幕| 99热国产这里只有精品6| 久久久久精品性色| 亚洲成人av在线免费| 搡老乐熟女国产| 精品亚洲成a人片在线观看| 一区二区三区精品91| 婷婷色综合www| 中文字幕人妻丝袜一区二区 | 五月天丁香电影| 久久久久国产网址| 久久热在线av| 精品人妻在线不人妻| 精品久久久久久电影网| 香蕉精品网在线| 免费观看在线日韩| 黄色怎么调成土黄色| 在线观看一区二区三区激情| 免费播放大片免费观看视频在线观看| 天堂俺去俺来也www色官网| 免费观看av网站的网址| 在线观看免费高清a一片| 成年女人在线观看亚洲视频| 十分钟在线观看高清视频www| 国产精品国产三级国产专区5o| 大陆偷拍与自拍| 国产人伦9x9x在线观看 | 国产亚洲最大av| 国产深夜福利视频在线观看| 香蕉国产在线看| 日韩欧美精品免费久久| 99国产精品免费福利视频| 五月伊人婷婷丁香| 国产精品免费视频内射| 久久99一区二区三区| 日日爽夜夜爽网站| 亚洲,欧美,日韩| 七月丁香在线播放| 晚上一个人看的免费电影| 中文天堂在线官网| 2021少妇久久久久久久久久久| 黄色一级大片看看| 丰满迷人的少妇在线观看| 日韩视频在线欧美| 一本色道久久久久久精品综合| 国产精品久久久久久久久免| 91国产中文字幕| 久久久久网色| 叶爱在线成人免费视频播放| 七月丁香在线播放| 色视频在线一区二区三区| 国产精品麻豆人妻色哟哟久久| 日韩中字成人| 黄色 视频免费看| 男女啪啪激烈高潮av片| 国产精品久久久久成人av| 国产在线免费精品| 久久久久久久精品精品| 亚洲国产精品999| 一边亲一边摸免费视频| 日本欧美视频一区| 国产成人免费观看mmmm| 国产乱人偷精品视频| 男人操女人黄网站| 免费看av在线观看网站| 国产精品偷伦视频观看了| 亚洲国产精品成人久久小说| 欧美最新免费一区二区三区| 少妇猛男粗大的猛烈进出视频| 国产精品不卡视频一区二区| 亚洲综合色网址| 精品亚洲乱码少妇综合久久| 五月伊人婷婷丁香| 香蕉精品网在线| 日产精品乱码卡一卡2卡三| 亚洲四区av| 十八禁网站网址无遮挡| 天堂8中文在线网| 久久久精品免费免费高清| 各种免费的搞黄视频| 久久久久久人妻| 两性夫妻黄色片| 久久久久国产精品人妻一区二区| 美女大奶头黄色视频| 日韩免费高清中文字幕av| 黄色配什么色好看| 午夜免费鲁丝| 人人妻人人爽人人添夜夜欢视频| 久久久精品94久久精品| av在线老鸭窝| 18禁裸乳无遮挡动漫免费视频| 纯流量卡能插随身wifi吗| 久久99蜜桃精品久久| 日韩 亚洲 欧美在线| 成年美女黄网站色视频大全免费| av国产久精品久网站免费入址| 国产欧美亚洲国产| 啦啦啦视频在线资源免费观看| 欧美激情 高清一区二区三区| www.精华液| 丝瓜视频免费看黄片| 午夜av观看不卡| 国产乱来视频区| 亚洲精品视频女| 欧美日韩精品网址| 欧美精品一区二区大全| 欧美日韩精品网址| a级毛片在线看网站| 欧美国产精品va在线观看不卡| 美女脱内裤让男人舔精品视频| 黄色怎么调成土黄色| 国产伦理片在线播放av一区| 中文字幕最新亚洲高清| 亚洲精华国产精华液的使用体验| 在线观看国产h片| 日韩欧美一区视频在线观看| 国产精品 欧美亚洲| 一级爰片在线观看| 国产亚洲欧美精品永久| 国产淫语在线视频| 国产精品欧美亚洲77777| 亚洲第一av免费看| 中文精品一卡2卡3卡4更新| 男女午夜视频在线观看| 中国三级夫妇交换| 欧美av亚洲av综合av国产av | 一区二区三区乱码不卡18| 国产成人精品在线电影| 咕卡用的链子| 免费观看在线日韩| 成年人午夜在线观看视频| 国产成人精品久久久久久| 一级毛片我不卡| 久久久久久久久久人人人人人人| 精品第一国产精品| 天天影视国产精品| 777米奇影视久久| 久久精品国产鲁丝片午夜精品| 天天躁夜夜躁狠狠久久av| 美国免费a级毛片| 亚洲成人av在线免费| 王馨瑶露胸无遮挡在线观看| 国产有黄有色有爽视频| 老司机亚洲免费影院| 国产成人精品久久久久久| 99久久精品国产国产毛片| 精品国产乱码久久久久久男人| 一区二区三区乱码不卡18| 99精国产麻豆久久婷婷| av不卡在线播放| 狠狠婷婷综合久久久久久88av| 亚洲综合精品二区| 国产免费福利视频在线观看| 欧美日韩av久久| 美女大奶头黄色视频| 人妻少妇偷人精品九色| 一级片免费观看大全| 777久久人妻少妇嫩草av网站| 亚洲国产精品一区二区三区在线| 国产 精品1| 中文天堂在线官网| 日韩一区二区三区影片| 制服诱惑二区| 久久久久久人妻| 国产精品秋霞免费鲁丝片| 国产在线一区二区三区精| 少妇人妻久久综合中文| 亚洲第一av免费看| 免费av中文字幕在线| 久久狼人影院| 亚洲精品国产色婷婷电影| kizo精华| 91在线精品国自产拍蜜月| 日本黄色日本黄色录像| 色播在线永久视频| 狠狠婷婷综合久久久久久88av| av不卡在线播放| 永久免费av网站大全| 777久久人妻少妇嫩草av网站| 日韩一本色道免费dvd| 秋霞在线观看毛片| 爱豆传媒免费全集在线观看| 精品亚洲成国产av| 国产成人精品婷婷| 蜜桃在线观看..| 青青草视频在线视频观看| 国产在线视频一区二区| 亚洲婷婷狠狠爱综合网| 亚洲欧美精品自产自拍| 少妇人妻精品综合一区二区| 大香蕉久久成人网| 天天躁夜夜躁狠狠躁躁| 18禁裸乳无遮挡动漫免费视频| 日本欧美视频一区| 国产人伦9x9x在线观看 | 9191精品国产免费久久| 亚洲欧美一区二区三区黑人 | 观看av在线不卡| 九九爱精品视频在线观看| 国产成人免费观看mmmm| 一区二区三区乱码不卡18| 老司机影院毛片| 午夜日韩欧美国产| 满18在线观看网站| 国产日韩欧美视频二区| 日韩伦理黄色片| 高清在线视频一区二区三区| 一级片'在线观看视频| 80岁老熟妇乱子伦牲交| 久久久久久久精品精品| 成人亚洲欧美一区二区av| 高清欧美精品videossex| 国产成人精品无人区| 免费黄色在线免费观看| 日韩中字成人| 亚洲少妇的诱惑av| 午夜久久久在线观看| 热99久久久久精品小说推荐| 国产综合精华液| 少妇人妻 视频| 妹子高潮喷水视频| 搡女人真爽免费视频火全软件| 男女午夜视频在线观看| 色哟哟·www| 国产毛片在线视频| 久久午夜福利片| 国产成人精品久久久久久| 国产精品免费视频内射| 69精品国产乱码久久久| 国产乱人偷精品视频| 啦啦啦在线观看免费高清www| 国产精品国产av在线观看| 人人澡人人妻人| 九草在线视频观看| 99久久精品国产国产毛片| a级毛片在线看网站| 另类精品久久| 久久精品国产自在天天线| 精品亚洲乱码少妇综合久久| 国产精品免费大片| 亚洲精品美女久久久久99蜜臀 | 99热网站在线观看| 九九爱精品视频在线观看| 日韩三级伦理在线观看| 免费看不卡的av| 国精品久久久久久国模美| 国产av精品麻豆| 人妻少妇偷人精品九色| 在线观看免费日韩欧美大片| 99久久人妻综合| 可以免费在线观看a视频的电影网站 | 久久精品国产亚洲av涩爱| 精品国产国语对白av| 制服丝袜香蕉在线| 国产乱来视频区| 久久久久精品久久久久真实原创| av在线老鸭窝| 久久ye,这里只有精品| 十八禁网站网址无遮挡| 亚洲成人手机| 永久免费av网站大全| 丰满饥渴人妻一区二区三| 一区二区av电影网| 在线观看免费视频网站a站| 制服人妻中文乱码| 90打野战视频偷拍视频| 99久国产av精品国产电影| 日本-黄色视频高清免费观看| 亚洲欧美日韩另类电影网站| 视频区图区小说| 国产免费又黄又爽又色| 两个人看的免费小视频| 最近中文字幕2019免费版| 国产精品99久久99久久久不卡 | 久久午夜综合久久蜜桃| 老汉色av国产亚洲站长工具| 精品卡一卡二卡四卡免费| 亚洲国产精品一区二区三区在线| 久久久久久免费高清国产稀缺| 日本vs欧美在线观看视频| 人妻 亚洲 视频| 国产av国产精品国产| 免费观看av网站的网址| 麻豆乱淫一区二区| 人人妻人人澡人人看| 欧美精品高潮呻吟av久久| 国产精品人妻久久久影院| 免费在线观看完整版高清| 亚洲欧美一区二区三区久久| 视频在线观看一区二区三区| 如何舔出高潮| 成年女人毛片免费观看观看9 | 婷婷色av中文字幕| av免费在线看不卡| 国产不卡av网站在线观看| 嫩草影院入口| 亚洲精品自拍成人| www.自偷自拍.com| 欧美中文综合在线视频| 伦理电影大哥的女人| 曰老女人黄片| 观看av在线不卡| 国产又爽黄色视频| 亚洲精品久久午夜乱码| 天堂中文最新版在线下载| 久久久久久免费高清国产稀缺| 国产97色在线日韩免费| 久久av网站| 夜夜骑夜夜射夜夜干| 91国产中文字幕| 老女人水多毛片| 久久热在线av| 免费观看在线日韩| 999精品在线视频| 美女国产视频在线观看| 蜜桃在线观看..| 一级毛片我不卡| 亚洲精品日韩在线中文字幕| 久久精品国产a三级三级三级| 97在线视频观看| 成人免费观看视频高清| 最近的中文字幕免费完整| 国产精品秋霞免费鲁丝片| 赤兔流量卡办理| 久久午夜福利片| 这个男人来自地球电影免费观看 | 在线天堂最新版资源| 久久精品久久久久久久性| 亚洲精品乱久久久久久| 亚洲精品美女久久久久99蜜臀 | 一区二区三区乱码不卡18| 久久99精品国语久久久| 男女午夜视频在线观看| 天堂8中文在线网| 午夜福利在线观看免费完整高清在| 国产无遮挡羞羞视频在线观看| 国产精品人妻久久久影院| 日本黄色日本黄色录像| 欧美日韩精品网址| 亚洲国产av影院在线观看| 日韩在线高清观看一区二区三区| 日韩免费高清中文字幕av| 国产一区二区三区av在线| 欧美97在线视频| 91精品国产国语对白视频| 久久免费观看电影| 亚洲少妇的诱惑av| 天堂8中文在线网| 国产福利在线免费观看视频| 三上悠亚av全集在线观看| 国产精品 欧美亚洲| 国产高清不卡午夜福利| 久久精品久久久久久久性| 777米奇影视久久| 亚洲中文av在线| 精品少妇一区二区三区视频日本电影 | 最新的欧美精品一区二区| 精品亚洲成国产av| 最近中文字幕高清免费大全6| 成人毛片a级毛片在线播放| 久久精品国产亚洲av天美| 在线观看国产h片| 午夜福利在线观看免费完整高清在| 国产精品蜜桃在线观看| www.熟女人妻精品国产| 精品国产露脸久久av麻豆| 老汉色∧v一级毛片| 国产精品免费大片| 成人亚洲欧美一区二区av| 国产无遮挡羞羞视频在线观看| 高清av免费在线| 国产高清不卡午夜福利| 免费看av在线观看网站| www.精华液| 国产精品av久久久久免费| 欧美亚洲日本最大视频资源| 免费久久久久久久精品成人欧美视频| 欧美最新免费一区二区三区| 妹子高潮喷水视频| 国产精品三级大全| 99国产综合亚洲精品| 黄色 视频免费看| 欧美少妇被猛烈插入视频| 精品国产乱码久久久久久男人| 久热久热在线精品观看| 一级毛片电影观看| av.在线天堂| 男人舔女人的私密视频| 成人漫画全彩无遮挡| 边亲边吃奶的免费视频| 十八禁高潮呻吟视频| 国产成人av激情在线播放| 老鸭窝网址在线观看| 国产亚洲精品第一综合不卡| 久久97久久精品| 欧美亚洲 丝袜 人妻 在线| 久久国产精品男人的天堂亚洲| 嫩草影院入口|