• <tr id="yyy80"></tr>
  • <sup id="yyy80"></sup>
  • <tfoot id="yyy80"><noscript id="yyy80"></noscript></tfoot>
  • 99热精品在线国产_美女午夜性视频免费_国产精品国产高清国产av_av欧美777_自拍偷自拍亚洲精品老妇_亚洲熟女精品中文字幕_www日本黄色视频网_国产精品野战在线观看 ?

    Deep Learning Based Attack Detection for Cyber-Physical System Cybersecurity: A Survey

    2022-01-26 00:35:22JunZhangLeiPanQingLongHanChaoChenShengWenandYangXiang
    IEEE/CAA Journal of Automatica Sinica 2022年3期

    Jun Zhang,,Lei Pan,,Qing-Long Han,,Chao Chen,,Sheng Wen,,and Yang Xiang,

    Abstract—With the booming of cyber attacks and cyber criminals against cyber-physical systems (CPSs),detecting these attacks remains challenging.It might be the worst of times,but it might be the best of times because of opportunities brought by machine learning (ML),in particular deep learning (DL).In general,DL delivers superior performance to ML because of its layered setting and its effective algorithm for extract useful information from training data.DL models are adopted quickly to cyber attacks against CPS systems.In this survey,a holistic view of recently proposed DL solutions is provided to cyber attack detection in the CPS context.A six-step DL driven methodology is provided to summarize and analyze the surveyed literature for applying DL methods to detect cyber attacks against CPS systems.The methodology includes CPS scenario analysis,cyber attack identification,ML problem formulation,DL model customization,data acquisition for training,and performance evaluation.The reviewed works indicate great potential to detect cyber attacks against CPS through DL modules.Moreover,excellent performance is achieved partly because of several highquality datasets that are readily available for public use.Furthermore,challenges,opportunities,and research trends are pointed out for future research.

    I.INTRODUCTION

    CYBER-physical systems (CPSs) suffer from cyber attacks when they are increasingly connected to the cyber space.According to [1] published in 2017,more than 30 surveys were published to cover the cybersecurity issue in the CPSs.Cyber attacks have become increasingly sophisticated and prevalent as automated attacking tools,and professional hacking groups have started to get involved.A successful cyber attack against a CPS may be disastrous,catastrophic,or even fatal [2]–[6].However,it is a challenge to defend against cyber attacks on CPSs.Many CPS systems lack cybersecurity mechanisms like message authentication,resulting in challenges to detect false data injection attacks.A lack of universal encryption,especially on the systems employing dated technologies,makes it challenging to defend against eavesdropping attacks.System states need to be referred to detect replay attacks.In addition,the use of dated technology in operation limits the choices of defenses to network traffic in most cases [7].

    Deep learning (DL) [8],[9] delivers superior performance to traditional machine learning (ML) solutions.Whenever there is adequate data,DL models almost deliver excellent results.However,DL models have been slowly applied to solve the CPS cybersecurity issue compared with other fields such as NLP,image processing,software vulnerability [10],[11],and many more [12]–[17].It is also observed that many DL models have been proposed in recent publications to detect CPS cyber attacks.A widely accepted view to explain the difficulty of detecting cyber attacks on CPSs was accredited to the degree of complexity when superposing cybersecurity over CPSs [2].

    There exist a few short-length survey papers on CPS cybersecurity [1],[2],[18],[19].Some papers investigated data-driven methods for detecting cyber attacks against CPS systems [18],[20].However,there is no detailed discussion on applying DL methods to detect CPS cyber attacks.A short survey was provided in [18] with a four-step framework to apply DL methods on CPS issues,including cybersecurity,adaptability,recoverability,and many more,without a specific focus on cybersecurity.Furthermore,most of the cited works in [18] were published between 2012 and 2016,but this survey includes most papers between 2017 and 2021.A survey of surveys was presented in [1] without relevance to DL models.A comprehensive survey on the cyber attacks against CPSs was presented in [20] without investigating the DL models.Various methods of detecting cyber attacks in the CPSs were summarized in [2] without using DL methods.A comprehensive list of CPS attacks and challenges were provided in [19] but overlooking ML,or DL approaches.A cybersecurity analysis framework was proposed in [21]without utilizing the rich sources of available data.A recently published survey in [22] presents cybersecurity control and state estimation from active and passive defence perspectives.

    Fig.1.The DL driven methodology for CPS cybersecurity considers the essential needs for training robust and usable DL models in the context of cyber attacks against the CPS systems.

    We aim to review current research works on the advances of DL driven solutions for detecting cyber attacks in the CPS domain.It provides an overview for readers to quickly understand and step into the field by following our six-step DL driven methodology.Our six-step methodology considers the complete cycle of DL application from broad scenarios to performance evaluation.This paper caters to researchers,practitioners,and students interested in building DL-based cybersecurity applications in CPSs.The key contributions of this survey are three-fold:

    1) We conduct an up-to-date review of detecting cyber attacks in CPSs using DL models and propose a six-step methodology to position and analyze the surveyed works.

    2) We provide an overview for the state-of-the-art solutions with preservation of technical details.

    3) Based on the methodology,we discuss the challenges and future research directions.

    The rest of this survey is organized as follows: Section II proposes a research methodology for deep learning driven CPS cybersecurity.Section III presents the reviews on stateof-the-art research.Section IV discusses the research challenges and future work.Finally,Section V concludes this survey.

    II.RESEARCH METHODOLOGY

    Our methodology represents a deep understanding of the surveyed papers.The process consists of six steps,including CPS scenario analysis,cyber attack identification,DL problem formulation,DL model construction,data acquisition,and performance evaluation.Fig.1 shows a process of detecting cyber attacks in the context of a CPS by using DL models.For example,a smart grid may suffer from erroneous controls derived by electric load forecasts [20],[23].Falsely injected messages containing maliciously crafted information need to be identified and eliminated before committing the prediction process.A stacked AutoEncoder(AE) proposed in [24] may serve as a reliable regressor to predict the energy load on the system.The chosen AutoEncoder was subsequently trained with sufficient simulation data.At last,the DL model delivered excellent prediction results with the mean absolute percentage error of 3.51% on annual predictions.

    A.Step I: CPS Scenario Analysis

    The normal operations of CPSs rely on several important factors,including dependability,real-time operation,fault tolerance,cybersecurity,and many more.We must consider these requirements holistically.Dependability consists of service availability and reliability to minimize the system downtime; real-time operation is a critical factor for maintaining the system operation when the inputs and environment rapidly change; fault tolerance requires that the critical components of the system have sufficient backups to prevent the system from shutting down; and cybersecurity requirements are becoming more and more prominent when many CPSs are connected to the cyber space to improve the quality of system control and the overall level of quality of service.According to Mitchellet al.[2],there are four primary categories of characteristics of CPS intrusion detection,including physical process monitoring,closed control loops,attack sophistication,and legacy technology.

    Physical process monitoring:Physical properties of a CPS should be constantly monitored to identify any anomalies of the system because many physical processes of the CPS follow the laws of physics.

    Closed control loops:CPS events are significantly more regular and predictable than user-triggered events because many CPS events are driven by the preset feedback-based controllers.

    Attack sophistication:Sophisticated cyber attacks are increasingly popular in the CPS context because the potentially huge payoff for a successful cyber attack may bring sensitive information,valuable intelligence for military or finance operations,and many more.

    Legacy technology:Legacy hardware commonly used in the CPSs cannot interact with software-defined control because of the existing mechanical and hydraulic control.

    Analyzing the characteristics of a CPS scenario will help craft an appropriate cybersecurity problem.The involvement of physical signals enriches the input variables and complicates the design of any security solutions for CPSs.Although the behaviors of simplified proof-of-concept systems are relatively regular and predictable,real-world systems often operate in a noisy environment with unprecedented cyber threats.

    B.Step II: Cyber Attack Identification

    Upon completion of identifying the CPS scenario,we need to define a set of appropriate cyber attacks associated with CPS characteristics.For example,we will have more confidence to detect the falsely injected network packets if physical processes of the CPS components are properly monitored; cyber attacks like replay attacks may be detected on a CPS with a closed control loop; unknown attacks and sophisticated attacks like web attacks need to be considered if there is any concern of attack sophistication; denial of service(DoS) attacks and replay attacks are more prevalent in the presence of legacy technology.

    Based on the surveyed articles,we identify many common cyber attacks.Some frequent cyber attacks against the industrial control network include false data injection attacks,DoS attacks,replay attacks,and alike; and some frequent cyber attacks against the software-based controllers with a centralized server include brute force attacks,botnets,web attacks,heartbleed attacks,infiltration attacks and many more.Effective and efficient detection of these cyber attacks can be leveraged by using DL models,so we will need to translate the cybersecurity problem to the ML domain.

    C.Step III: ML Problem Formulation

    After aligning the cyber attacks to the CPS characteristics,the research problem can be translated to the ML/DL domain.ML is defined in [25] as “A computer program is said to learn from experienceEwith respect to some class of tasksTand performance measureP,if its performance at tasks inT,as measured byP,improves with experienceE.” DL is referred to in [8] as solving a complex problem by using a hierarchy of more straightforward concepts without too much human intervention.The definition of ML is general,and we will implement an ML solution in multiple steps.In this step,we need to define the taskT,including classification,clustering,regression,etc.A classification task requires that the trained model allocates its output to a pre-defined set of “classes”which could be the specific cyber attack categories; a clustering task often requires that the trained model allocates its output to a few “clusters” which could indicate normal traffic or attack traffic; a regression task is also known as a prediction task which requires the trained model to predict some numerical values.For example,a classification problem was found in [26] to differentiate cyber attack types; a clustering problem was found in [27] to separate covert messages from the normal messages; and a regression problem was set in [24] to predict the electric load in a smart grid.The choice of the ML tasks will impact the construction of the DL models.

    D.Step IV: DL Model Customization

    The DL model is constructed by selecting an architecture suitable for the research problem and optimizing parameters.The choice of DL models should be made according to actual needs.For example,autoencoders are good at translating the input data so that they are suitable for learning the representations of the data often required in prediction or regression tasks [24]; convolution networks (CNNs) and other models are usually used in classification tasks [28].

    The configuration of the chosen DL model also depends on the available data.A DL model with a large number of neurons per layer will almost always require more data than a DL model with the same design but a few neurons per layer.Some trade-offs can also be made by stacking more hidden layers inside the DL model instead of expanding the layer size.The ways and insights of the customizing model can be explored based on a thorough understanding of DL algorithms and CPS cybersecurity data.Furthermore,we can achieve improvement at various levels by combining the choice of DL models with a specific research problem.

    E.Step V: Data Acquisition for Training

    Data acquisition is a critical step for training DL models.The quality and quantity of data determine the effectiveness of solving the research problem.Also,data can serve as the source for setting up ground truth and affect the prediction model’s performance.One of the simplest methods to collect data is through simulation.This method is often used to generate datasets for power grids such as IEEE 9-bus,14-bus,30-bus,and 118-bus systems in Matlab.The other method relies on several existing datasets harvested by other researchers.These datasets include the SWaT dataset1http://itrust.sutd.edu.sg/dataset/SWaT,the SCADA IDS dataset2https://sites.google.com/a/uah.edu/tommy-morris-uah/ics-data-sets,the CICIDS2017 dataset3https://sites.google.com/a/uah.edu/tommy-morris-uah/ics-data-sets,the UNSWNB15 dataset4https://www.unsw.adfa.edu.au/unsw-canberra-cyber/cybersecurity/ADFANB15-Datasets/,and the KDD99 Cup dataset5http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html.

    Different cyber attacks were included in the datasets:

    1) The SWaT dataset contained eleven days of network traffic collected from a scaled-down water treatment plant.And there were no attacks during the first seven days.It includes 36 types of cyber attacks that are most commonly seen in today’s CPS systems.

    2) The SCADA IDS dataset contained network traffic logs of a SCADA IDS system.It includes seven types of cyber attacks — injection random response packets,hide the real state of the controlled process,inject malicious state commands,inject malicious parameter commands,inject malicious function code commands,DoS attack,and recon attack.

    3) The CICIDS2017 dataset [29] contained network traffic logs collected from an industrial control system.It includes six types of cyber attacks — brute force attacks,botnet,DoS attack,web attack,heartbleed attack,and infiltration attack.

    4) The Bot-IoT dataset [30] contained network traffic logs collected from an IoT setup.It includes three types of cyber attacks — infiltration,DoS attack,and information theft.

    5) The power system dataset [31] contained the network traffic collected from a power grid.It includes three cyber attacks — data injection,remote command injection,and replay attacks.

    6) The UNSW-NB15 dataset [32] contained the network traffic information extracted form a 100 GB packet capture dump.It includes nine cyber attacks — DoS attacks,exploits,recon attacks,worms,fuzzers,web penetration attacks,backdoors,shellcode,and generic attacks against block ciphers.

    7) The KDD99 Cup dataset [33] contained the network traffic information presented at the ACM SIGKDD conference in 1999.It includes four cyber attacks — DoS attacks,unauthorized accesses,privilege escalation,and probing attacks.

    F.Step VI: Performance Evaluation

    The last step is used to determine whether the DL model meets our expected objectives through performance evaluation.The performance is usually measured according to various metrics.We divide the performance metrics into two categories according to the tasks: 1) For prediction or regression tasks,a number of error metrics are used to measure the performance,including mean absolute error(MAE),mean relative error (MRE),root of mean squared error (RMSE),and mean absolute percentage error (MAPE).2) For classification or clustering tasks,there are a few standard metrics,including accuracy,recall,precision,false positive rate (FPR),F1 score.And occasionally,graphical plots like receive operating characteristic (ROC) curves are used by plotting TPR asy-axis and FPR asx-axis to depict the trade-offs between benefits and costs.Finally,area under ROC curve (auROC) is used to indicate the cumulative strength of a particular ROC curve.

    In many cases,FPR poses challenges for the DL models because the false alarms almost always result in excessive costs associated with manual verification.And it is always challenging to detect the rare or even unknown attacks as proven in [34] so that most of the surveyed literature aimed to maximize the TPR while minimizing the FPR.On the other hand,the error rate can be tolerated more generously in the regression task than in the classification task.By leveraging comprehensive evaluation metrics,we can decide whether the outputs of a specific DL model are satisfactory.Whenever there are unsatisfactory results,the process should be repeated with proper adjustments.

    III.CPS CYBERSECURITY WITH DEEP NEURAL MODELS

    This Section surveys the relevant literature of detecting cyber attacks in the context of CPSs by following the research methodology described in Fig.1.In particular,the body of the literature is divided into two parts according to the DL architectures,which will be elaborated below.

    A.Representation Learning for Attack Detection

    An AutoEncoder-based (AE) model was proposed in [26] to preserve privacy information in the context of smart power networks.Data privacy violations are becoming more and more popular in smart power networks.It is challenging to defend against inference attacks,because the smart power networks represent the CPS characteristics of physical process monitoring,closed control loops,attack sophistication,and legacy technology.The research problem of defending against inference attacks was translated into a classification problem in the ML domain.A Variational AutoEncoder (VAE) was proposed to provide transformed features for the ultimate classification task and transform raw data into an encoded format for preventing inference attacks.A VAE is a feedforward model used for encoding an input into new data codes using a set of weighted parameters.The VAE consisted of one input layer,four hidden layers,and one output layer.The transformed data from the output layer were written to the database for publication.Two datasets were used to evaluate the VAE,i.e.,the power system dataset [31] and the UNSWNB15 dataset [32].The Power system dataset is a multi-class dataset involving 37 scenarios that include 8 natural events,28 intrusive events,and 1 no event; and the UNSW-NB15 dataset includes a combination of current normal and attack records.300,000 random samples of legitimate and attack observations were chosen from each dataset for assessing the performance of the proposed framework.Although the VAE was only employed as a part of the intrusion detection system,its strength was demonstrated while transforming complex data into a simple form.The VAE achieved 0.921 for accuracy and 0.005 for loss on the power system dataset,and 0.998 for accuracy and 0.0001 for loss on the UNSW-NB15 dataset.

    An AE-based solution was proposed in [35] to detect various cyber attacks in the context of industrial control networks.There exist many kinds of cyber attacks when control networks are connected to the internet.The research problem generally reflects the CPS characteristics of attack sophistication.And it was translated to a classification problem in the ML domain.Hence,a 7-layer AE consisted of an input layer,four hidden layers,and an output layer.The input layer had 41 units corresponding to the feature space’s dimension,and the output layer had five units corresponding to the five types of network traffic.In particular,the last hidden layer was a softmax layer to provide the stability of the model.The AE was trained using the NSL-KDD dataset [33].As an early study,the proposed AE suffered low performance in detecting small classes like probe attack and remote attack.The stacked AE achieved 0.978 for accuracy over the five categories.The model achieved an F1 score of 0.9683.

    An AE-based model was proposed in [24] to detect cyber attacks in the context of smart grids.One big challenge is a large number of control parameters.The smart power networks represent the CPS characteristics of physical process monitoring and legacy technology.The smart grid’s essential controller is based on state estimation,so the lower and upper bounds of each state variable need to be predicted as accurately as possible.Hence,this research problem was translated into a regression problem in the ML domain.A stacked AE (SAE) was proposed to process the smart grid data.The SAE consisted of an input layer,three vanilla AEs,and a logistic regressor as the output layer.The SAE was trained with simulated data representing IEEE 9-bus,14-bus,30-bus,and 118-bus systems.Overall,the SAE in this study achieved excellent results in predicting the electric load forecast.The mean absolute percentage error (MAPE) was used to evaluate the SAE’s accuracy.And the SAE achieved a MAPE of 3.51% on an annual prediction and outperformed the baseline models like SVM and BP.Despite the SAE model’s simplicity,the empirical studies showed its applicability and consistency in performing load forecasts.

    Another AE-based model was proposed in [36] to detect Phasor measurement unit data manipulation attacks (PDMAs)in smart grids.PDMAs are challenging to be detected because of the similarity between PDMAs and man-in-the-middle attacks with infiltration of communication networks.This problem represents the CPS characteristics of physical process monitoring and attack sophistication.The main idea was to detect anomalies based on the normal operation patterns from the data collected from the PMUs with PDMA-free measurements in a distributed manner.Hence,the research problem was translated to a regression problem in the ML domain.A deep AE (DAE) was constructed by stacking four RBM models.Training the deep AE required multiple stages by fine-tuning the intermediate RBMs.The input layer took 108 numerical features,and the output layer is a regressor.The dataset was collected from a simulated IEEE 9-bus system and had 250 000 records.The deep AE was trained by using 200 000 benign records,20 000 records were reserved as the validation dataset,and the testing dataset consisted of 30 000 samples with half from attack records.The studies showed that the deep AE outperformed the baseline models like OCSVM,C4.5,MLP,SVM,and kNN.The DAE achieved 0.941 for accuracy,0.996 for precision,0.886 for recall,and 0.9038 for F1 score.Despite the success of using the deep AE in this study,it is challenging to obtain benign data from real-world power networks,and new methods may need to be explored.

    An AE-based model was proposed in [37] to detect attacks against physical measurements in the context of smart grids.This problem represents the CPS characteristics of physical process monitoring and legacy technology.It is challenging to derive useful features for intrusion detection in a noisy environment in a real-world factory.Hence,the research problem was translated into a classification problem in the ML domain.A stacked denoising AE (SDAE) was proposed to learn the advanced features from the input data.And the learned features were fed to an ELM for classification.Simulated data from gas turbines were collected and used to train the model.The proposed model achieved excellent results with an FPR of 0.000006,which was significantly below the required FPR of 0.01.The SDAE was used as a part of the IDS model but demonstrated its strength in extracting useful features to represent physical measurements from a noisy environment.

    An AE-based model was proposed in [28] to detect cyber attacks in the context of the industrial control systems.This problem represents the CPS characteristics of physical process monitoring,attack sophistication,and legacy technology.Hence,the problem was translated into a classification problem in the ML domain.An AE was proposed to extract features for a 1D CNN classifier.The AE consisted of five layers,including an input layer,a corruption layer applying Gaussian noise to the input,a fully connected layer with an activation function,an encoder layer,a decoding layer as the output layer to generate the extracted feature.The SWaT dataset was used to train the model.In particular,the training time for the AE was less than half-second,which was significantly faster than the 1D CNN model.The AE achieved 0.890 for precision,0.827 for recall,and 0.844 for F1 score.In summary,the AE model was validated as a powerful and efficient method to extract useful features.

    An LSTM autoencoder architecture was proposed to detect cyber attacks in the context of the autonomous vehicles (AVs)[38].AVs are linked together by using communication technologies,and thus are vulnerable to network attacks,such as Denial of Service,replay ans spoofing attacks.Such attacks can be inferred from network traffic.Authors designed an LSTM autoencoder to detect these cyber attacks.Statistical features from network traffic were extract to represent the activities of AVs.The designed neural network architecture was consisted of two types layers,LSTM and fully connected layer.A number of LSTM layers were used to encode the representation of the transformed likelihood stream.Then the reconstructed output was produced by the fully connected layer.Two datasets,i.e.,Car Hacking dataset and UNSWNB15 were used to evaluate the proposed scheme.In particular,the proposed LSTM based autoencoder achieved 0.99 for precision,1.0 for recall,and 0.99 for F1 score in the Car Hacking dataset.While on UNSW-NB15 dataset,the proposed scheme achieved 0.1 for precision,0.97 for recall,and 0.98 for F1 score.In a word,this work can successfully detect multiple types of attack vectors.

    Remark 1:Research works employing AE-based architecture were summarized in Table I.Most of them focused on smart grids or power network systems.Due to the difficulties in smart grids’ control systems,most AE models were used to learn the useful features of an intrusion detection system or predict the electric load as an indicator of cyber attacks.Moreover,the AE models were relatively small in size,so that they could be trained in a short amount of time.

    B.Cyber Recognition with Deep Learning Methods

    1) Cybersecurity Pattern Recognition with Deep Neural Networks (DNNs):A DNN-based model was proposed in [39]to learn the communication patterns between electronics control units (ECUs) in the context of in-vehicular network security.The security of communication messages among ECUs is vital because a group of ECUs can control and monitor a vehicle’s status during a maneuver.It is challenging to ensure cybersecurity because most communications between ECUs are through the controller area network protocol,which has no support for authentication or integrity check.Specifically,fake packets injected into the open communication channel through the controller area network protocol pose severe cybersecurity risks.Detecting the fabricated or modified packets in the vehicular setup needs to meet the requirements of physical process monitoring and legacy technologies.This intrusion detection problem was translated into a binary classification problem in the ML domain.That is,statistical features were extracted from highdimensional CAN packet data through a dimension reduction process to represent the normal and attack packets.A 5-layered DNN model was constructed based on a standard DBN model by adding a binary classification layer as the final output layer.The DBN’s coefficient weights were determined through an unsupervised pre-training process,but the final DNN model was trained with a bottom-up supervised manner.During each simulation round,a total of 200 000 packets were generated by the Open Car Test-bed and Network Experiments(OCTANE) generator.A 70:30 split was made to divide training and testing sets.Many experiments were conducted by varying the layers of the DNN model from 5 to 11 to investigate the trade-offs between performance and efficiency.The empirical results demonstrated the effectiveness of the proposed DNN model while comparing it with ANN and SVM.The best performance was achieved as 0.978 foraccuracy,0.016 for false positive rate,and 0.028 for false negative rate.Given the detection ratio of over 99%,the proposed DNN model showed good potentials to detect fake packets on vehicular networks despite that the DNN models’efficiency with more than five layers needed to be improved to meet the real-time requirements.

    TABLE IRESEARCH WORKS EMPLOYING AUTOENCODERS (AES)

    Another DNN-based model was proposed in [40] to learn the network traffic patterns in the electric power grid context.The cybersecurity of an electric power grid largely depends on state estimation underpinning critical control processes for the grid.It is challenging to detect false data injection attacks against the state estimation because a skilled cyber attacker may disguise the injected data stealthily with the inside knowledge of system topology.Such successful attacks may blackout an entire region due to the falsely impacted state estimation because the injected data value is progressively added to the legitimate signal and the Gaussian noise values.Detecting the injected data fed to the state estimation model needs to meet the requirements of physical process monitoring and closed control loops in a power grid.Hence,this intrusion detection problem was translated into a binary classification problem with the objective function of simultaneously minimizing the number of false positives and false negatives.A series of measurement vectors were created for a specific time slot so that a compromised vector contains any injected component corresponding to a false data inject attack.Four variations of DNN models were constructed with different settings — 1 or 3 hidden layers,100 or 150 neurons per hidden layer,whether to use L1 regularization.The DNN models were trained with the standard stochastic gradient descent approach using the back-propagation method.The activation function was tanh.The most accurate DNN model was also the most complex among all the four DNN models.In terms of accuracy,the DNN model of 3 hidden layers with 150 neurons on each layer without L1 regularization outperformed generalized linear models,gradient boosting machines,a distributed random forest classifier,and the other three DNN models.The best performance was 0.9802 for precision,0.9895 for recall,0.9852 for F1 score,and a low false alarm rate of 0.1840.The proposed DNN model demonstrated the effectiveness of a simulated IEEE 14-bus power grid without testing realistic datasets generated by Real-time Digital Simulation (RTDS) and physical testbeds.

    A DNN-based model was proposed in [41],[42] to detect the anomalies in the context of secure water treatment(SWaT).The SWaT system represents many challenges of securing CPSs,including physical process monitoring,closed control loops,attack sophistication,and legacy technologies.A divide-and-conquer strategy was employed in this scenario by separating different sets of sensors and actuators into groups according to their functionalities.Thereafter,the high dimension and complexity of detecting anomalies were mitigated.Data points with a low probability were regarded as outliers because common processes in the system usually generated the normal data points.The proposed DNN model included an LSTM layer and 100 intermediate layers.The LSTM layer predicted the actuator’s position based on its historical positions.Each hidden layer was fully connected to an output layer with a bi-linear function.The cost function was defined by the cross-entropy of the real probability distribution and the predicted probability distribution.The dataset was the log entries collected from 51 sensors and actuators of a testbed for 11 days.According to [41],[42],the DNN detected 13 of the total 36 scenarios,and the DNN model achieved 0.98295 for precision,0.67847 for recall,and 0.80281 for F1 score.However,it took two weeks to train the DNN model and 8 hours to complete testing the data.Due to its inefficiency,DNN has limited use for real-world applications.

    A DNN-based model was proposed in [27] to detect covert message transmission in the context of a chemical process plant.The covert messages containing critical control information exfiltrated from the actuators can be used to detect anomaly operations on hardware devices.Because the covert channel was established without any modification to the system,conducting data analytics on the covert channel had to involve CPS characteristics,including physical process monitoring and closed control loops.It is challenging for simplistic solutions to detect the messages transmitted via a covert channel when analog emission from the physical instrument was used to disguise the existence of the messages.Hence,the problem of detecting covert messages was translated to a clustering problem.A 10-layers DNN received the inputs from the digitized audio samples and produced binary outputs to indicate whether there was a covert message transmission or not.The ten layers included two dropout layers,four linear layers,three ReLU activation functions,and a tanh activation function.The DNN model was trained using Adam optimizer to maintain the learning quality.The dataset consisted of 9 minutes of audio recordings recorded by a Hardware-In-The-Loop (HITL) simulator.Due to the nature of analog signals,the performance of the DNN model was measured by an accuracy of 0.95 on testing data in online operation.The successful application of the DNN model opened many opportunities to develop and deploy real-time monitoring mechanisms over critical actuators using audio sampling and covert channels.

    2) Cybersecurity Pattern Recognition with Convolutional Neural Networks (CNNs):A CNN-based model was proposed in [43] to detect cyber attacks in the context of an industrial water treatment plant.Because the SWaT dataset was used in this study,it inherited many CPS characteristics,including physical process monitoring,closed control loops,attack sophistication,and legacy technologies.An anomaly detection approach was used for improving the detection rate of the cyber attacks on the SWaT dataset.A 1D CNN model was constructed by stacking a set of 1D convolution layer,a ReLU function,and a 1D max-pooling layer before applying to flatten,dropout,and a final fully connected layer.Among all the 36 different attack scenarios,the CNN model successfully detected 31.The CNN model achieved 0.912 for precision,0.861 for recall,and 0.886 for F1 score.When using 8 layers,the 1D CNN model reached 0.967 as the AUC value with only 15 535 parameters.This model’s training time and testing time for one epoch were 88 seconds and 47 seconds,respectively.Based on the positive results of this empirical study,the proposed CNN model achieved excellent results in detecting cyber attacks on the SWaT dataset and demonstrated good potentials in real-world deployment.

    A CNN-based model was proposed in [44] to detect message injection attacks in the vehicular networks.Due to the lack of security protections to the controller area network(CAN),network packets with malicious contents can be easily injected to the CAN bus resulting in gaining control of the vehicle.The detection of malicious packets on a physical vehicle needs to be conducted according to the CPS requirements.Two Raspberry Pi devices acting as a listener and an attacker were connected to an operational passenger vehicle via the OBD-II port to validate the model’s effectiveness.The attacker had four attacking modes — DoS attack,fuzzy attack,drive gear spoofing attack,and engine RPM gauge spoofing attack.It is challenging to effectively and efficiently detect all of the attacks in real-time.A deep CNN model named Inception-ResNet was employed with two blocks of convolution layers,pooling layers,a fully connected layer,and a softmax layer before generating the final output.The activation function was set to the ReLU function.The whole dataset was split into four sub-datasets according to the attack scenarios.The CNN model outperformed other classifiers,including LSTM,ANN,SVM,kNN,NB,and Decision Trees.Each attack lasted for 3 to 5 seconds during the four recording sessions of forty minutes each.Thus,each dataset contained 300 injection attack instances.The number of messages in the four attacking scenarios are 3 078 250 (DoS normal),587 521 (DoS attack),3 347 013 (fuzzy normal),491 847 (fuzzy attack),2 766 522 (gear normal),597 252 (gear spoofing),2 290 185 (RPM normal),and 654 897 (RPM spoofing).False negative rate (FNR) and error rate (ER) were used to measure the CNN model’s performance.In terms of FNR,the CNN model achieved 0.06,0.07,0.10,and 0.24 for gear spoofing,RPM spoofing,DoS attack,and fuzzy attack,respectively; in terms of ER,the CNN model achieved 0.03,0.04,0.05,and 0.18 for DoS attack,RPM spoofing,gear spoofing,and fuzzy attack,respectively.The empirical studies showed that the CNN model was a promising technique for detecting false message injection attacks to vehicular networks.

    A federated deep learning scheme (DeepFed) in [45],to detect cyber threats targeting industrial CPSs.Li et al.firstly designed a CNN-GRU based intrusion detection model.Second,they took account into the federated learning scenario,and built a framework to allow multiple industrial CPSs to build an intrusion detection model together.Lastly,they applied a secure communication protocol based on Paillier cryptosystem to preserve the privacy of model parameters.

    Industrial CPSs are not only targeted by traditional cyber threats,such as DoS attacks,but also by some cyber threats which are customized to industrial systems,such as response injection attacks.Furthermore,in the federation based CPS cyber threats detection framework,eavesdropping attacks on data sources and model parameters are emerging.Thus,the authors proposed DeepFed,which was based on CNN-GRU to detect such threats.The model architecture was consisted of a CNN module,a GRU module,an MLP module,and a softmax layer.To evaluate the performance of DeepFed,they ran experiments on a real-world dataset collected from a gas pipelining system,with 80% splited for training and the rest 20% for testing.The performance varied when different number of local agents (K) were in consideration.If K = 3,DeepFed could achieve an accuracy,precision,recall,F-score of 99.20%,98.86%,97.34%,and 98.08%.Overall,all metrics could reach over 97%.The experiments demonstrated the effectiveness of DeepFed to detect different types of cyber threats to industrial CPSs.

    3) Cybersecurity Pattern Recognition with Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM)Models:An LSTM-based model was proposed in [46] to identify anomaly sensor behaviors in the water treatment plant context.Detecting the cyber attacks in a real plant is challenging because of the complex setups of the sensors and actuators representing many CPS characteristics,including physical process monitoring,closed control loops,attack sophistication,and legacy technologies.Specifically,the anomaly behaviors and the sensors under attack needed to be identified at the same time.The LSTM model using a cumulative sum was proposed to learn the sensors’ behaviors to achieve a low false positive rate.The LSTM model consisted of three LSTM stacks with 100 hidden units each.The cumulative sum of the sequence predictions was introduced to reduce false positives because of their capabilities of indicating small deviations over time.And its loss function was a mean-square loss function.Ten attack scenarios of the SWaT dataset was used for training the LSTM model.The attack scenarios were made up of six single-stage-single-point attacks,two single-stage-multi-point attacks,one multi-stage-single-point attack,and one multistage-multi-point attack.It took about 24 hours to train the LSTM model,which is a big drawback.Nine out of ten attack scenarios were detected without listing detailed results for specific performance metrics.This work was an early study of applying DL models on the SWaT dataset with only limited success.Nevertheless,it showed good potentials for applying DL models for detecting cyber attacks against CPSs.

    An LSTM-based model was proposed in [47] to learn the traffic pattern generated by unknown or zero-day attacks in the context of a control system for a gas pipeline.Detecting unknown or zero-day attacks is challenging because most intrusion detectors on industrial control systems are manually configured for specific protocols and systems.A hybrid strategy of combining packet contents and temporal dependencies and a fast signature-based Bloom filter were used to triage anomaly traffic based on the network packets’contents.A subsequent and slow LSTM model was dedicated to capturing the unknown or zero-day attacks based on the time-series information.The proposed LSTM model consisted of two LSTM layers and a softmax layer.Each LSTM layer had 256 fully connected neuron cells,and the output softmax layer had 613 bits to match the length of signatures used by the Bloom filter.A dataset of network logs collected from the gas pipeline was used to train the model,where there were seven types of cyber attacks with a total of 214 580 normal network packets and 60 048 attack packets.Combining a fast Bloom filter and a slow but powerful LSTM model demonstrated its success in detecting cyber attacks in a small SCADA system.Regarding the detection ratio (recall),the hybrid model outperformed the other six classifiers for six out of seven attacks,including Bloom filter,Bayesian network,SVDD,isolation forest,PCA-SVD,and Gaussian mixture model.The proposed model also achieved 0.92 for accuracy,0.94 for precision,0.78 for recall,and 0.85 for F1 score.This study showed the promising application of the LSTM model to detect time-series anomalies supplemented by an efficient and lean model like Bloom filter.

    An RNN-based model was proposed in [48] to detect various cyber attacks in the context of smart grids.There are various cyber attacks against smart grids,such as DoS attacks,data infiltration,and so on.Detecting all the attacks on a large scale smart grid network is challenging because of the CPS characteristics in terms of attack sophistication and legacy technology.Hence,a vanilla RNN model was trained by using the truncated backpropagation through time (BPTT)algorithm.Three datasets were fed to the RNN model to cover a wide range of cyber attacks.In particular,the CICIDS2017 dataset [29] included brute force attacks,botnets,DoS attacks,web attacks,heartbleed attacks,and infiltration attacks; the Bot-IoT dataset [30] consisted of infiltration,DoS attack,and information theft; and the power system dataset [31] included data injection,remote command injection,were used to replay attacks.The CICIDS2017 dataset contains 2 830 743 records,the Bot-IoT dataset 73 360 900 records,and the power system dataset 78 404 records.Experiments were conducted on the three datasets separately,and the results showed that the RNN model outperformed the benchmark classifiers like SVM,random forest,and NB in terms of mean false positive rate —0.00986 for the CICIDS2017 dataset,0.01281 for the Bot-IoT dataset,and 0.03986 for the power system dataset.In terms of accuracy,the RNN model achieved 0.98941 for the CICIDS2017 dataset,0.99912 for the Bot-IoT dataset,and 0.96882 for the power system dataset,respectively.Although the proposed vanilla RNN performed well across the dataset,its integration with the blockchain component of the DeepCoin system for detecting fraudulent transactions remains unclear but may inspire further research works to use blockchain technologies together with DL models.

    An LSTM-based model was proposed in [49] to detect anomalous automatic dependent surveillance-broadcast (ADSB) messages transmitted between airplanes and control towers.A typical ADS-B message includes an aircraft’s flight information,such as flight number,speed,GPS coordinates,altitude,and many more.Due to historical design and implementation,the ADS-B system is an open messaging system without authentication or encryption.Therefore,it is challenging to defend ADS-B systems against malicious message injection attacks in the aviation industry because the ADS-B messages are transmitted very frequently at an average rate of 4.2 messages per second.Detecting the spoofed ADS-B messages represents the CPS characteristics of legacy technology.Hence,the problem was translated into an unsupervised anomaly detection problem in the ML domain.An LSTM model was proposed to capture the message sequences to detect anomalies according to estimated credibility scores.The LSTM model consisted of two vanilla RNN models stacked as an encoder-decoder paradigm.The LSTM was trained to reconstruct sequences of benign messages with minimal errors,where the reconstruction error score was calculated by using the cosine similarity.A largescale flight tracking dataset namedFlightradar24was used to train the LSTM model using flight data collected from 13 international airports.The LSTM achieved 1.00 as recall,0.03 as FPR,and 0.99 as TPR.The LSTM outperformed the baseline anomaly detection methods,including HMM-GMM,one-class SVM (OCSVM),local outlier factor (LOF),isolation forest (IF),and DBSTREAM.In particular,the spoof messages through RND and ROUTE attacks were detected almost instantaneously,but the SHIFT Down and SHIFT Up attacks were detected with some delay.The LSTM model achieved excellent results in detecting the spoofing messages.

    An LSTM-based model was proposed in [50] to predict multimedia data requests transmitted in the cyber-physical industrial networks.Multimedia data as the carrier for many cyber attacks are manifold,including image files,audio files,and many more.The prediction of multimedia data through a cache resource allocation system became a new proposal to defend the underlying network,representing the CPS characteristics of attack sophistication.Hence,this research problem was translated as a regression problem in the ML domain.A vanilla LSTM model was proposed to capture the spatio-temporal relations of the multimedia contents.Specifically,the LSTM consisted of 3 layers where the input layer’s size was equal to the number of multimedia types,and the hidden layer’s size was half of the input size.A two-yearlong network traffic logs collected from a factory-intensive area in Tianjin,China,were considered the normal traffic of the dataset; all cyber attacks were manually added to the dataset.And various ratios of training data and testing data were used to train the LSTM model.The empirical studies showed that the LSTM accurately predicted the multimedia contents.The LSTM outperformed three baseline models:SVM,ANN,and RNN.The performance was measured in MAE,MRE,and RMSE,where the LSTM achieved 3.9857 for MAE,0.0553 for MRE,and 4.6166 for RMSE.Despite the LSTM’s excellent performance,predicting multimedia contents in the industrial network remains open.

    4) Cybersecurity Pattern Recognition with Deep Belief Network (DBN):A DBN-based model was proposed in [51] to detect false data injection attacks in the context of energy internet.It is challenging to detect stealthy cyber attacks due to many control signals and meter reading data on the energy internet.And this represents the CPS characteristics of physical process monitoring,closed control loops,and legacy technology.In this study,the detection problem was formulated as a dual bi-level programming problem with the upper and lower bounds because the variations of the electric loads can be predicted.And the prediction problem is translated as a regression problem in the ML domain.A DBN model was trained to forecast electric load.The DBN was made of three stacked RBM models forming six layers — an input layer,four hidden layers,and a logistic regression layer as the output layer.The DBN was trained with the data collected from simulated IEEE 14- and 118-bus systems.An overall error rate was used to measure the DBN model’s performance,and the DBN achieved a 2.73% error rate that was almost 3% lower than the benchmark model SVM.In this study,the DBN used only as a forecasting component of the intrusion detection model is a good example of trading off between the DL model and other programming solutions.

    A DBN-based model was proposed in [52] to detect false data injection attacks in electric power networks.The detection of false data injection attacks represents the CPS characteristics of physical process monitoring,closed control loops,and legacy technology.The research problem was translated into a classification problem in the ML domain.The detection accuracy was investigated through the features automatically generated by DL models.Specifically,a DBN model was proposed as a baseline approach to compare with an RNN model and a graph neural network (GNN) model.The DBN was created by using the default settings of Tensorflow package without optimization.The data was collected from two simulated systems,IEEE 30-bus and 118-bus.The DBN achieved 0.9939 for precision and 0.98231 for recall on the IEEE 30-bus,which was almost identical to the GNN model and slightly higher than the RNN model.However,on the IEEE 118-bus,the DBN model’s performance was consistently better than the RNN model and comparable to the graph neuron network model.Overall,the DBN model shows consistency,and the GNN model shows some promising future.

    Remark 2:Table II lists thirteen different DL models for detecting various cyber attacks across various CPS scenarios.Among the models,four were DNN-based,two CNN-based,five RNN-based,and two DBN-based.It took longer time and more computational resources to train some models than others because of the nature of the architectures and parameter settings.In general,CNNs,DNNs,and DBNs were faster to train than RNNs and LSTMs.But the slower models like LSTMs or even an embedded LSTM layer had their merits in forecasting electric power load soon.On the contrary,DNNs,CNNs,and DBNs provided more powerful capabilities to detect anomalies.Last but not least,DNNs earned their popularity partly because of their simple setup and various hyperparameter fine-tuning techniques.

    IV.CHALLENGES AND FUTURE OPPORTUNITIES

    Six potential areas are depicted in Fig.2 where challenges and new research directions may arise.These six areas correspond to the six steps of our research methodology,as shown in Fig.1.Our research methodology helps provide an overview of the research literature and extract important elements for comparative analysis.The analysis results underpin research challenges and opportunities in the near future.

    A.New CPS Cybersecurity Scenarios

    The papers studied the communication networks in CPS scenarios.The majority of the surveyed publications investigated the CPS scenarios in water treatment plants or smart grids,which accounted for thirteen out of twenty surveyed papers.Among the remaining six papers,two studied vehicle networks,one on generic industrial control networks,one on chemical process plant,one on aviation communication networks,and one on gas pipeline controllers.The imbalanced topics suggest that CPS scenarios were underrepresented.

    Applying DL to the 21st-century manufacturing industry is an emerging topic.DL has been used to detect flaws anddefects during the manufacturing process of complicated items such as semiconductor [53],and 3D printing [54].They were excluded because no cybersecurity issues were covered.A hybrid model was proposed in [53] to detect wafer defect patterns during the semiconductor fabrication process,and an AE-based model was proposed in [54] to detect defects in the 3D printed objects.These two works primarily considered the causes of defects as a production issue or a modeling issue without the presence of cyber attacks.However,cyber attacks and threats may exist in the manufacturer’s network or in the cloud server where the design models are stored.Moreover,blockchain technologies have been studied in two of the surveyed papers [26],[48] in smart grids for providing additional cybersecurity and privacy guarantee.We anticipate that blockchain will be studied much more widely together with CPS.And the diversity and development of CPS scenarios will require intensive studies on cybersecurity.

    TABLE IICYBERSECURITY PATTERN RECOGNITION WITH DEEP LEARNING

    B.Identification of New Cyber Threats

    Almost all the surveyed papers studied false data injection attacks.The detection of stealthy false data injection attacks is challenging because of the large amount of noise produced in the CPS and the lack of cybersecurity mechanisms to authenticate devices and messages transmitted across the network.There are a few types of false injection attacks,depending on the attacker’s information and goals.For example,no advanced information is required for launching DoS attacks; only recorded packets are required for replay attacks; scanning tools are required for probing attacks;automated tools are required for fuzzy attacks.The effective and efficient analysis of the network traffic is crucial for defending against these attacks [55]–[57].

    Furthermore,CPS cybersecurity is much broader than cyber attacks against CPS systems.Detecting the cyber attacks that originated in the cyber space and penetrated the physical domain remains a great challenge.For example,the Stuxnet attack [58] utilized many cyber attack elements like system vulnerabilities,network sniffing,and many more,to exploit the kinetic system.Such attacks may cause significant damage and loss to the physical infrastructure.There are proposals to mitigate these sophisticated attacks by monitoring the CPS system’s cyber component and the physics component.However,it remains an open problem to correlate the interdependent monitoring mechanism in a distributed and realtime manner.

    Fig.2.Research directions for DL driven CPS cybersecurity.

    Considering the characteristics of CPS cybersecurity,it needs to explore three topics — advanced persistent threats,insider attacks,and cyber incident forecasts.Advanced persistent threats (APTs) quickly appear due to the availability of attack tools and techniques and are difficult to defend against because of their frequently updated signatures and evolving behaviors based on reconnaissance results [59].Insiders are very challenging to defend against because their specialized knowledge can be utilized to circumvent security checkpoints deployed on the perimeters of the critical infrastructure [60].Despite its potential high false positives,cyber incident forecasting is valuable to prepare early for fortifying strong defense and improve the overall cyber resilience [61].Due to available big data,many aspects of these three topics can be significantly advanced and improved by adopting ML and DL models.In particular,new insights and knowledge can be obtained via visualization,and deep analysis.We expect that emerging cyber attacks will precede the defense mechanisms,but the risk can be mitigated through the data-driven approach.

    C.Adopting New ML/DL Paradigms

    All the surveyed papers followed traditional ML paradigms,including supervised and unsupervised learning.Apart from four papers that examined regression problems and three papers on clustering problems,the remaining papers studied classification problems.The dominating use of the supervised learning paradigm reflected the importance of the well-labeled data.In particular,network packets were labeled as normal or attack traffic,and the attack types often were differentiated.Such reliance on labeled data restricted the wide adoption of ML or DL methods.We advocate for researchers and practitioners to try new ML/DL paradigms.These new paradigms include reinforcement learning and self-supervised learning,together with improving the model’s explainability.Instead of relying on learning the records,reinforcement learning focuses on experience [62].It is very suitable in isolated environments where many CPSs are currently deployed.For example,a reinforcement learning model was constructed in [63] to predict the normal driving behaviors.Deep reinforcement learning was applied to build human-level intelligence [64].Deep reinforcement learning may have huge potentials in fighting against cyber attacks on CPSs.Selfsupervised learning is a relatively new paradigm to generate data labels automatically.Self-supervised learning is a special kind of supervised learning without the need for the manual labeling process.In theory,self-supervised learning is ideal for changing environments where zero-day attacks and unknown attacks exist because the relationship between input data is deeply analyzed.A popular and successful model is BERT [65] for deriving deep bidirectional representations from unlabeled data.Because of BERT’s huge success in NLP and many other domains,we anticipate that self-supervised learning will prosper in the CPS domain because DL models generally suffer from poor explainability [66].We know how well the DL model works but do not know why the model works.Explainability is important for many CPS problems when we need to improve the system.Fortunately,tools like LEMNA [67] are available to explain DL-based cybersecurity applications.LEMNA can identify the critical features for a trained DL model like RNN and LSTM.We are optimistic to predict that the DL models will be more and more explainable when new tools and techniques are invented and used.

    D.Defending the Trained DL Models

    No surveyed works are considered to defend the trained DL models against various attacks.However,we would like to highlight the importance of defending the trained DL models because of the computational expenses to train the DL models,the important roles of the trained models,and potential dangers if the DL models are compromised.Due to the hunger for training samples,the DL models are sometimes trained with data from untrustworthy sources.Thus,adversarial attacks are prevalent because of linear behavior in highdimensional space [68].For example,Android HIV was proposed in [69] to automatically generate adversarial Android malware that the existing detectors failed to detect.

    Adversarial attacks are categorized into two types,including evasion attacks and poisoning attacks [70].Evasion attacks aim to evade a trained classifier by altering input data,and poisoning attacks aim to affect the trained classifier by injecting poisoning samples to the training dataset.These attacks differ by their time of occurrence.Poisoning attacks usually occur during the training phase,but evasion attacks occur during the test phase.Much research efforts have been devoted to fighting against data poisoning attacks,including robust optimization,adding some adversarial samples to training data,blind-spot removal,examining decision boundaries,and many more.There are many strategies to counter poisoning attacks,such as outlier removal,data sensitization,frequent model retraining,classifier ensembles,randomizing the classifier’s output,and many more.Last but not least,model stealing attacks were proven in [71] as a big threat to the trained model.The attackers can obtain sufficient information to mimic an ML or DL model by launching many queries and aggregating the results.The extracted information can be used to build a mimicking model for the attacker to find possible evasion attacks.We strongly advocate that cyber defense should be conducted as soon as possible due to the unawareness of adversarial attacks in the CPS scenarios.

    E.Enriching CPS Cybersecurity Datasets

    Among the surveyed papers,datasets collected in the field dominated the simulation with a 14:6 ratio.Simulated data were investigated in the two CPS scenarios — smart grids and vehicular networks.In smart grids,Matlab was the only choice for simulating electric load in five papers; and the OCTANE simulator was used in one paper on vehicular networks.However,there is a significant risk of solely relying on proprietary products like Matlab because the availability of such products may be discontinued unprecedently.On the other hand,field data is independent of the simulation platform and offers researchers good flexibility.Among the papers using field data,five papers chose the SWaT dataset,two papers the CICIDS2017 dataset,and the rest seven papers different datasets.The SWaT dataset dominated the field data category for a few reasons: 1) The network traffic data were continuously collected for 11 days from the control networks and from the sensors of a physical testbed,2) the traffic with and without attack were chronologically separated for easy use,and 3) there were 36 attack scenarios against different components of the testbed.To our surprise,the NSL-KDD dataset [33],also known as KDD99-cup,was studied in one surveyed paper despite its age of 20+ years.Using such a dated dataset may cause people to draw biased conclusions because many cyber attacks were not included in the NSLKDD dataset.Therefore,we recommend the researchers to use datasets like the UNSW-NB15 dataset [32] where recent cyber attacks were present.

    Moreover,new datasets will always be valuable and appreciated.Ideally,the new datasets are open-sourced field data collected from physical testbeds.Several CPS testbeds are proposed to facilitate detecting cyber attacks,such as[72]–[74].The recent trend of increasing interest in building CPS testbeds may benefit researchers to collect high-quality attack and defense data.The new datasets should be large enough to exploit DL models’ power,and both new and old cyber attacks should be included because cyber attacks evolve quickly.If labeling data is challenging,then chronologically separating the attacks from the normal traffic is a feasible idea.Artificially blending the data entries representing attacks into a set of normal traffic records should be avoided because the simple data augmentation method does not consider feasibility,attacking sequences,and possible changes of correlations.To benefit the advancement of research and knowledge,we strongly encourage more and more highquality datasets to be made available to the community.

    F.Improving the Model Evaluation

    Standard performance metrics were used in most of the surveyed papers.False positives were investigated,along with accuracy and error rates.It is proven in [34] that it is significantly more difficult to detect the rarely occurred attacks than the common ones derived by the Bayesian laws.In real-world CPSs,cyber attacks may rarely occur,so the DL models trained in the lab settings may be invalid.Since most papers did not investigate the impact of the imbalanced data between normal and attack traffic,the empirical results may be substantially biased or inflated.Cross comparisons in [75]showed that the precision-recall curve (PRC) and the area under precision-recall curve (auPRC) were more resilient to imbalanced data than ROC and auROC.Therefore,new studies should consider reporting PRC and auPRC.

    Furthermore,time decay should be considered in future studies because each trained ML or DL model’s performance will inevitably degrade over time.When the cyber attacks rapidly evolve,the models trained with old data will struggle with detecting new attacks.A time decay metric was proposed in [75] to evaluate a trained model’s performance loss.By studying the time decay,we will be able to decide when the model needs to be retrained.We strongly hope to see future work similar to [75] in the context of CPS and cyber attacks.Once the in-depth knowledge is developed and gained,we may expect to mitigate the risk of CPS cyber attacks.

    V.CONCLUSION

    This survey provides a current view of detecting cyber attacks in the CPSs.A six-step DL driven methodology is proposed to summarize and analyze the twenty recently published papers in this survey.Specifically,a panoramic view is obtained through inspecting the CPS scenarios,identifying cybersecurity problems,translating the research problem to the ML/DL domain,constructing the DL model,preparing datasets,and finally evaluating the model.Cyber attacks persist as an ongoing and prominent threat to the security and safety of the CPSs.The reviewed works show great potential to exploit CPS cyber data through DL models because of their promising performances.The excellent performance is achieved partly because of several high-quality datasets that are readily available for public use.In addition to following the success of current research,we also identified promising research topics,including integration with blockchain,detection of advanced persistent threats,adopting new ML and DL paradigms,prevention of adversarial and model extraction attacks,enriching datasets,and applications of additional performance metrics.We are optimistic and confident that the research in this field will flourish.

    成人黄色视频免费在线看| 精品国产露脸久久av麻豆| 成人18禁高潮啪啪吃奶动态图 | av线在线观看网站| 啦啦啦在线观看免费高清www| 中国美白少妇内射xxxbb| 亚洲精品第二区| 男女国产视频网站| 亚洲国产精品999| 午夜激情福利司机影院| 精品久久久久久久久亚洲| 中国美白少妇内射xxxbb| 在线免费观看不下载黄p国产| 午夜久久久在线观看| av.在线天堂| 丝瓜视频免费看黄片| 成人无遮挡网站| 免费人妻精品一区二区三区视频| 一级,二级,三级黄色视频| 亚洲精品视频女| 亚洲精品亚洲一区二区| 2022亚洲国产成人精品| 一级,二级,三级黄色视频| 亚洲精品久久久久久婷婷小说| 国产黄色免费在线视频| 国产高清不卡午夜福利| av有码第一页| 国产熟女欧美一区二区| 国产有黄有色有爽视频| 天堂中文最新版在线下载| 天天躁夜夜躁狠狠久久av| 国产精品成人在线| 少妇被粗大猛烈的视频| 久久久久久久久大av| 久久人人爽人人爽人人片va| 极品人妻少妇av视频| 国产精品麻豆人妻色哟哟久久| 十八禁高潮呻吟视频 | 人妻 亚洲 视频| 在线亚洲精品国产二区图片欧美 | 日韩制服骚丝袜av| 人妻一区二区av| 一个人看视频在线观看www免费| 国产精品蜜桃在线观看| 亚洲图色成人| 男人狂女人下面高潮的视频| 极品少妇高潮喷水抽搐| 大片免费播放器 马上看| 亚洲人与动物交配视频| 热re99久久精品国产66热6| 国产在线一区二区三区精| 国产精品麻豆人妻色哟哟久久| 丰满饥渴人妻一区二区三| 97在线人人人人妻| 亚洲国产精品一区三区| 在线观看免费日韩欧美大片 | 高清黄色对白视频在线免费看 | 日日爽夜夜爽网站| 国产免费一区二区三区四区乱码| 久久久久精品性色| 99久久精品国产国产毛片| 精品午夜福利在线看| 国产精品国产三级国产av玫瑰| www.色视频.com| 日本vs欧美在线观看视频 | 99热全是精品| 免费看日本二区| 少妇人妻精品综合一区二区| 天堂俺去俺来也www色官网| 欧美激情极品国产一区二区三区 | 这个男人来自地球电影免费观看 | 性色avwww在线观看| 亚洲四区av| 免费黄色在线免费观看| 91精品国产九色| 熟妇人妻不卡中文字幕| 亚洲国产精品国产精品| 熟女人妻精品中文字幕| 少妇人妻精品综合一区二区| 51国产日韩欧美| 国产精品国产三级国产专区5o| 特大巨黑吊av在线直播| 国产精品蜜桃在线观看| 午夜福利影视在线免费观看| 国产在视频线精品| 久久久久网色| 国产中年淑女户外野战色| 国产精品熟女久久久久浪| 99热6这里只有精品| 男人狂女人下面高潮的视频| videos熟女内射| 极品少妇高潮喷水抽搐| 99re6热这里在线精品视频| 色网站视频免费| 亚洲精品aⅴ在线观看| 欧美精品一区二区大全| 久久精品国产a三级三级三级| 久久狼人影院| 中文字幕制服av| 免费看不卡的av| 插逼视频在线观看| 亚洲精品乱码久久久久久按摩| 国产熟女午夜一区二区三区 | 国产精品无大码| 国产日韩欧美在线精品| 亚洲三级黄色毛片| 亚洲精品中文字幕在线视频 | 99热这里只有精品一区| 在线观看人妻少妇| 国产成人精品一,二区| 91成人精品电影| 国产成人精品一,二区| 久久久精品免费免费高清| 久久这里有精品视频免费| 久久av网站| 五月开心婷婷网| 日韩欧美 国产精品| 男人舔奶头视频| 亚洲精品,欧美精品| www.色视频.com| 日本猛色少妇xxxxx猛交久久| 男女国产视频网站| 久久久国产一区二区| 亚洲美女视频黄频| 亚洲av.av天堂| 18禁裸乳无遮挡动漫免费视频| 精品午夜福利在线看| 亚洲国产欧美在线一区| 精品久久久久久电影网| 各种免费的搞黄视频| 亚洲av.av天堂| 亚洲精品国产色婷婷电影| 成人18禁高潮啪啪吃奶动态图 | 午夜免费男女啪啪视频观看| 精品少妇黑人巨大在线播放| av福利片在线观看| 亚洲av欧美aⅴ国产| 深夜a级毛片| 亚洲av中文av极速乱| 丝袜在线中文字幕| 日本黄色片子视频| 亚洲美女搞黄在线观看| av视频免费观看在线观看| 日韩一区二区视频免费看| 中文字幕久久专区| 高清在线视频一区二区三区| 欧美 日韩 精品 国产| 国产精品一区二区三区四区免费观看| 曰老女人黄片| 青青草视频在线视频观看| 日本色播在线视频| 久久久久久久久久成人| 欧美成人精品欧美一级黄| 一级毛片我不卡| 国产精品欧美亚洲77777| 欧美亚洲 丝袜 人妻 在线| 久久鲁丝午夜福利片| 精品一区二区三卡| 日本黄大片高清| 国产视频内射| 国产在线一区二区三区精| 国产精品一区二区在线观看99| 久久久久精品久久久久真实原创| 中文字幕av电影在线播放| 九九爱精品视频在线观看| 久久久国产精品麻豆| 少妇人妻久久综合中文| 一本大道久久a久久精品| 免费黄频网站在线观看国产| 国产成人精品久久久久久| 九色成人免费人妻av| 人妻少妇偷人精品九色| 亚洲精华国产精华液的使用体验| 美女主播在线视频| 久久午夜福利片| 欧美一级a爱片免费观看看| 国产免费一级a男人的天堂| 精品一区二区免费观看| 国产一区二区在线观看日韩| 三级国产精品片| 亚洲欧美中文字幕日韩二区| 91成人精品电影| 新久久久久国产一级毛片| 国产高清不卡午夜福利| 人人妻人人澡人人看| 日本av手机在线免费观看| 99热6这里只有精品| 久久精品夜色国产| 国产老妇伦熟女老妇高清| 久久国产亚洲av麻豆专区| 午夜免费鲁丝| 国产精品女同一区二区软件| 国产亚洲午夜精品一区二区久久| 天堂8中文在线网| 日日摸夜夜添夜夜爱| 麻豆成人午夜福利视频| 亚洲精品中文字幕在线视频 | 韩国av在线不卡| 美女中出高潮动态图| 国产免费一级a男人的天堂| 亚洲av电影在线观看一区二区三区| 99热这里只有是精品在线观看| 成人亚洲精品一区在线观看| 国产精品三级大全| 国产精品伦人一区二区| 一级毛片黄色毛片免费观看视频| 成人漫画全彩无遮挡| 精品99又大又爽又粗少妇毛片| 国产免费福利视频在线观看| 日本爱情动作片www.在线观看| 91成人精品电影| 最新的欧美精品一区二区| 一级片'在线观看视频| 日韩免费高清中文字幕av| 精品久久久噜噜| 国产精品成人在线| 国产免费一区二区三区四区乱码| 亚洲国产成人一精品久久久| 你懂的网址亚洲精品在线观看| 十分钟在线观看高清视频www | 少妇的逼水好多| 精华霜和精华液先用哪个| 亚洲性久久影院| 日本wwww免费看| 国产精品99久久99久久久不卡 | 国产白丝娇喘喷水9色精品| 大码成人一级视频| 婷婷色av中文字幕| 日本-黄色视频高清免费观看| 多毛熟女@视频| 久久久久久久大尺度免费视频| 日本免费在线观看一区| 91久久精品国产一区二区三区| 女的被弄到高潮叫床怎么办| 国内少妇人妻偷人精品xxx网站| 国产av国产精品国产| 少妇精品久久久久久久| 国产熟女午夜一区二区三区 | 久久久久久久久久久丰满| 男的添女的下面高潮视频| 黄色怎么调成土黄色| 亚洲av成人精品一区久久| 国产精品国产三级专区第一集| 99久久综合免费| 秋霞伦理黄片| 亚洲av男天堂| 美女视频免费永久观看网站| 国产免费福利视频在线观看| 日本av手机在线免费观看| 99热这里只有是精品50| 日本-黄色视频高清免费观看| 国产男人的电影天堂91| 蜜桃久久精品国产亚洲av| 22中文网久久字幕| 午夜老司机福利剧场| kizo精华| 国产精品秋霞免费鲁丝片| 51国产日韩欧美| 色94色欧美一区二区| 免费av中文字幕在线| 99热这里只有精品一区| 国产亚洲5aaaaa淫片| 能在线免费看毛片的网站| 精品一区二区三卡| 最新中文字幕久久久久| 亚洲精品第二区| 人人妻人人澡人人爽人人夜夜| 国产视频首页在线观看| 老女人水多毛片| 国产亚洲5aaaaa淫片| 中文字幕精品免费在线观看视频 | 亚洲图色成人| 街头女战士在线观看网站| 大香蕉97超碰在线| 亚洲欧美成人精品一区二区| 日韩中文字幕视频在线看片| 在线亚洲精品国产二区图片欧美 | 亚洲精品色激情综合| 热re99久久精品国产66热6| 日本免费在线观看一区| av黄色大香蕉| 久久久久久久久久久久大奶| 日韩欧美精品免费久久| 国产欧美亚洲国产| 免费av中文字幕在线| av在线app专区| 街头女战士在线观看网站| 色婷婷久久久亚洲欧美| 看十八女毛片水多多多| 久久综合国产亚洲精品| 成人午夜精彩视频在线观看| 极品人妻少妇av视频| 国产精品国产三级国产专区5o| 中文字幕人妻丝袜制服| 国产精品久久久久久av不卡| 久久精品国产亚洲av涩爱| 在线观看三级黄色| 国产乱人偷精品视频| 深夜a级毛片| 国产精品99久久99久久久不卡 | 在线观看av片永久免费下载| 一级毛片我不卡| 国产探花极品一区二区| 多毛熟女@视频| 精品亚洲成a人片在线观看| 亚洲av欧美aⅴ国产| 在线观看美女被高潮喷水网站| 99热这里只有精品一区| 国产无遮挡羞羞视频在线观看| 亚洲精品一区蜜桃| 日本免费在线观看一区| 男女边吃奶边做爰视频| 久久精品夜色国产| 亚洲久久久国产精品| 在线观看免费日韩欧美大片 | 大片电影免费在线观看免费| 国产精品久久久久成人av| 日本欧美视频一区| 亚洲精品aⅴ在线观看| 亚洲精品久久午夜乱码| 777米奇影视久久| 99九九在线精品视频 | 91精品一卡2卡3卡4卡| av福利片在线观看| 久久国产乱子免费精品| 久久6这里有精品| 欧美成人午夜免费资源| 亚洲成人av在线免费| 精品久久久久久久久亚洲| 日韩欧美精品免费久久| 亚洲国产毛片av蜜桃av| 欧美xxⅹ黑人| 97在线视频观看| 丰满饥渴人妻一区二区三| 欧美+日韩+精品| 一区在线观看完整版| 啦啦啦中文免费视频观看日本| 极品教师在线视频| 久久av网站| 日韩精品有码人妻一区| 王馨瑶露胸无遮挡在线观看| 欧美3d第一页| 国产在线视频一区二区| 精品久久久久久电影网| 在现免费观看毛片| 麻豆成人av视频| 日本vs欧美在线观看视频 | 99热网站在线观看| 亚洲国产成人一精品久久久| 黄片无遮挡物在线观看| 99热全是精品| 汤姆久久久久久久影院中文字幕| 少妇人妻精品综合一区二区| 91久久精品电影网| 色婷婷久久久亚洲欧美| 我的老师免费观看完整版| 国模一区二区三区四区视频| 大陆偷拍与自拍| 国产精品久久久久久av不卡| 亚洲av二区三区四区| 嫩草影院新地址| 人妻一区二区av| 久久久久人妻精品一区果冻| 国产成人午夜福利电影在线观看| 激情五月婷婷亚洲| 午夜免费观看性视频| 国产一区亚洲一区在线观看| tube8黄色片| 少妇人妻久久综合中文| 精品亚洲乱码少妇综合久久| av免费在线看不卡| 久久久久久久亚洲中文字幕| 亚洲精品色激情综合| videos熟女内射| 日本wwww免费看| 自线自在国产av| 日韩伦理黄色片| 18禁动态无遮挡网站| av免费观看日本| 欧美3d第一页| 观看av在线不卡| 丝袜在线中文字幕| 又大又黄又爽视频免费| 熟妇人妻不卡中文字幕| www.色视频.com| 97在线人人人人妻| 日韩三级伦理在线观看| av在线老鸭窝| 国产一区亚洲一区在线观看| 日韩电影二区| 亚洲精品国产成人久久av| 秋霞伦理黄片| 久久久久人妻精品一区果冻| 亚洲久久久国产精品| 色视频在线一区二区三区| 亚洲久久久国产精品| 色视频在线一区二区三区| 精品少妇黑人巨大在线播放| 国产成人免费无遮挡视频| 国产成人freesex在线| 视频中文字幕在线观看| 欧美日本中文国产一区发布| 国产精品久久久久久av不卡| 国产av码专区亚洲av| 丰满少妇做爰视频| 欧美xxⅹ黑人| 日韩 亚洲 欧美在线| 男女边摸边吃奶| 免费在线观看成人毛片| 黄片无遮挡物在线观看| 国产欧美日韩一区二区三区在线 | 成人漫画全彩无遮挡| 熟女av电影| 丰满迷人的少妇在线观看| 国产精品不卡视频一区二区| 久久99热这里只频精品6学生| 久久久久国产网址| 毛片一级片免费看久久久久| 国产精品久久久久久久电影| 亚洲国产精品成人久久小说| 汤姆久久久久久久影院中文字幕| 一级毛片我不卡| 熟女电影av网| 曰老女人黄片| 欧美亚洲 丝袜 人妻 在线| 韩国高清视频一区二区三区| 观看av在线不卡| 成年av动漫网址| 人人妻人人澡人人看| 亚洲精品一区蜜桃| 精品午夜福利在线看| 一级毛片aaaaaa免费看小| 高清av免费在线| 少妇的逼水好多| 国产日韩一区二区三区精品不卡 | 人妻一区二区av| 国产黄片视频在线免费观看| 国产精品.久久久| 欧美日韩国产mv在线观看视频| 你懂的网址亚洲精品在线观看| 一本一本综合久久| 日韩一本色道免费dvd| 国产日韩欧美视频二区| 日本av免费视频播放| 如何舔出高潮| 久久精品国产鲁丝片午夜精品| 极品人妻少妇av视频| 亚洲国产毛片av蜜桃av| 久久久精品94久久精品| 高清视频免费观看一区二区| 91精品伊人久久大香线蕉| 欧美最新免费一区二区三区| 欧美 亚洲 国产 日韩一| 欧美精品人与动牲交sv欧美| 性色av一级| 最黄视频免费看| 国产乱人偷精品视频| av又黄又爽大尺度在线免费看| 国产伦精品一区二区三区四那| 久久久久人妻精品一区果冻| 国产精品一区二区三区四区免费观看| 国产在视频线精品| 男女啪啪激烈高潮av片| 男人狂女人下面高潮的视频| 亚洲怡红院男人天堂| 国产精品麻豆人妻色哟哟久久| 自拍偷自拍亚洲精品老妇| 麻豆成人av视频| 久久久国产一区二区| 一级av片app| 人人妻人人澡人人爽人人夜夜| 最近最新中文字幕免费大全7| 另类精品久久| 黑人巨大精品欧美一区二区蜜桃 | 91精品国产国语对白视频| 国产伦精品一区二区三区四那| 少妇人妻 视频| 久久人人爽人人片av| 欧美区成人在线视频| 久久6这里有精品| 国产毛片在线视频| 看免费成人av毛片| 国产精品秋霞免费鲁丝片| 久久久久久久大尺度免费视频| 男女边摸边吃奶| 狂野欧美激情性bbbbbb| 久久99一区二区三区| 大香蕉97超碰在线| 久久国产精品大桥未久av | 亚洲精品国产av成人精品| a级片在线免费高清观看视频| 看非洲黑人一级黄片| 亚洲精品第二区| 亚洲成人av在线免费| 成人国产麻豆网| 水蜜桃什么品种好| 亚洲成色77777| 777米奇影视久久| 伊人久久国产一区二区| 国产免费一级a男人的天堂| 毛片一级片免费看久久久久| 久久亚洲国产成人精品v| 亚洲一级一片aⅴ在线观看| 亚洲精品中文字幕在线视频 | 久久久国产一区二区| 日日摸夜夜添夜夜添av毛片| 精品国产国语对白av| 边亲边吃奶的免费视频| 妹子高潮喷水视频| 我的女老师完整版在线观看| 国产成人午夜福利电影在线观看| 美女福利国产在线| 中文天堂在线官网| 如何舔出高潮| 大码成人一级视频| 国产精品不卡视频一区二区| 国产免费一区二区三区四区乱码| 在线观看av片永久免费下载| 午夜精品国产一区二区电影| 五月玫瑰六月丁香| 内射极品少妇av片p| 一区二区三区免费毛片| 在线观看av片永久免费下载| 国产91av在线免费观看| 国产精品免费大片| 国产在线免费精品| 亚洲欧美中文字幕日韩二区| 最黄视频免费看| 夜夜爽夜夜爽视频| 日韩成人av中文字幕在线观看| 人体艺术视频欧美日本| 麻豆乱淫一区二区| 日本vs欧美在线观看视频 | 久久99热这里只频精品6学生| 全区人妻精品视频| 少妇熟女欧美另类| 色婷婷av一区二区三区视频| 国产av一区二区精品久久| 亚洲色图综合在线观看| 我的老师免费观看完整版| 嘟嘟电影网在线观看| 成人国产av品久久久| 日韩免费高清中文字幕av| 婷婷色麻豆天堂久久| 久久精品国产a三级三级三级| 97在线视频观看| 日韩熟女老妇一区二区性免费视频| 中文乱码字字幕精品一区二区三区| 日本色播在线视频| 欧美97在线视频| 久久久久久久精品精品| 天堂中文最新版在线下载| 国产无遮挡羞羞视频在线观看| 99久久中文字幕三级久久日本| 久久久欧美国产精品| 在线观看免费高清a一片| 国产永久视频网站| 狠狠精品人妻久久久久久综合| 中文精品一卡2卡3卡4更新| 国产一区二区三区综合在线观看 | 国产视频首页在线观看| 久久久久久久久久人人人人人人| 麻豆乱淫一区二区| 中国三级夫妇交换| 欧美+日韩+精品| 国产精品熟女久久久久浪| 哪个播放器可以免费观看大片| 夜夜爽夜夜爽视频| 高清不卡的av网站| 国产永久视频网站| 亚洲国产精品国产精品| 成人18禁高潮啪啪吃奶动态图 | 久久99一区二区三区| 国产精品蜜桃在线观看| 亚洲内射少妇av| 久久久国产精品麻豆| 在线免费观看不下载黄p国产| 少妇人妻 视频| 国产成人午夜福利电影在线观看| 日韩欧美精品免费久久| 黄片无遮挡物在线观看| 黄色欧美视频在线观看| 久久 成人 亚洲| 日韩av在线免费看完整版不卡| 十八禁高潮呻吟视频 | 久久久久久久久久久免费av| 在线免费观看不下载黄p国产| 亚洲av男天堂| 亚洲久久久国产精品| 国产老妇伦熟女老妇高清| 免费播放大片免费观看视频在线观看| 亚洲人成网站在线观看播放| 国语对白做爰xxxⅹ性视频网站| 欧美日韩在线观看h| 大又大粗又爽又黄少妇毛片口| 高清不卡的av网站| 亚洲精品亚洲一区二区| 日本91视频免费播放| 久久99热这里只频精品6学生| 中文天堂在线官网| 精品亚洲乱码少妇综合久久| 伦理电影免费视频| 黄色一级大片看看| 99九九在线精品视频 | 国产亚洲av片在线观看秒播厂| 成人黄色视频免费在线看| 亚洲人成网站在线播| 一本—道久久a久久精品蜜桃钙片| 国产精品国产av在线观看| 午夜福利视频精品| 久久99一区二区三区| 亚洲精品中文字幕在线视频 | 成人无遮挡网站| 天天躁夜夜躁狠狠久久av| 中文字幕制服av|