Multiclass Classification for Cyber Threats Detection on Twitter

Adnan Hussein and Abdulwahab Ali Almazroi

1College of Computer Science and Engineering,Department of Computer Science,AL-Ahgaff University,Mukalla,Yemen

2College of Computing and Information Technology at Khulais,Department of Information Technology,University of Jeddah,Jeddah,Saudi Arabia

ABSTRACT The advances in technology increase the number of internet systems usage.As a result,cybersecurity issues have become more common.Cyber threats are one of the main problems in the area of cybersecurity.However,detecting cybersecurity threats is not a trivial task and thus is the center of focus for many researchers due to its importance.This study aims to analyze Twitter data to detect cyber threats using a multiclass classification approach.The data is passed through different tasks to prepare it for the analysis.Term Frequency and Inverse Document Frequency(TFIDF)features are extracted to vectorize the cleaned data and several machine learning algorithms are used to classify the Twitter posts into multiple classes of cyber threats.The results are evaluated using different metrics including precision,recall,F-score,and accuracy.This work contributes to the cyber security research area.The experiments revealed the promised results of the analysis using the Random Forest(RF)algorithm with(F-score=81%).This result outperformed the existing studies in the field of cyber threat detection and showed the importance of detecting cyber threats in social media posts.There is a need for more investigation in the field of multiclass classification to achieve more accurate results.In the future,this study suggests applying different data representations for the feature extraction other than TF-IDF such as Word2Vec,and adding a new phase for feature selection to select the optimum features subset to achieve higher accuracy of the detection process.

KEYWORDS Cybersecurity;cyber threat detection;artificial intelligence;machine learning;Twitter

1 Introduction

With the advancement in technology and electronic systems,people’s lives have become almost completely dependent on technology because of the ease and speed of information transmission[1].On the other hand,there are some challenges and issues that have emerged with this advancement in technology.One of these challenges is detecting and mitigating security threats that try to penetrate the cyber systems and manipulate or destroy the system to achieve illegal goals[2,3].Recently,these challenges have increased,and there is a need to stop these attacks and threats.Cybersecurity threat detection is a field of study that detects cyber threats to stop them and mitigate them[4].

Recently,there is integrity between several kinds of sciences to solve the issues and improve performance [5].Machine learning is one of the promising fields in artificial intelligence science,which is changed many human life aspects by integrating it into other fields.This integrity creates a notable advancement in many technological aspects of human life.The use of artificial intelligence and machine learning in cybersecurity and the detection of cyber threats is important to be studied and investigated [3].Some recently published research has indicated that the use of artificial intelligence techniques in cybersecurity has contributed to solving multiple problems and contributed to improving the current security systems[6].Thus,there is a need to expand the research in this area and investigate more in this field.

Social networks such as Twitter,Facebook,Instagram,and others have contributed significantly in the last few decades to connecting the world and communities.With social networks,dissemination of news and information has become easy,cheap,and very fast,an event occurs in the West at a specific time,and people in the East see it and interact with it at the same moment of occurrence.This is considered a revolution in the world of data transmission[1].There are many people with bad intentions trying to use these platforms to spread conflict and inconvenience,as well as to destabilize the security of others on multiple levels,including the individual,the community,and the level of companies or countries.Confronting such threats and risks requires reliable studies that detect these threats before they spread,expose the identity of the attackers,and stop them thus limiting the spread of threats.

This study uses artificial intelligence and machine learning techniques to detect cybersecurity threats using social networking data taken from Twitter.The next section includes an explanation of the research literature.After that,the research methodology pipeline is explained in detail for each different practical stage for the data analysis.The following section includes a discussion of the experiment results and finally the conclusion and future work suggestions.

2 Literature Review

Machine learning is a subfield of artificial intelligence which focuses on using data and algorithms to simulate how humans learn.The main purpose of using machine learning in different systems is to increase and refine the outcomes of these systems.There are three main approaches to machine learning algorithms:supervised machine learning,unsupervised machine learning,and reinforcement machine learning[3].

Recently,machine learning has been integrated with other sciences and it has achieved a notable improvement in several fields such as healthcare sciences,military sciences,economic sciences,and so on.Cybersecurity is one of the fields that is concerned with defending against the threats and attacks of the cyber systems,machine learning is used to help in several cybersecurity tasks such as detection,mitigation,and prevention to build strong and accurate cybersecurity systems.In literature,there are a few works have been published to tackle the integration between cybersecurity and machine learning and there is a need to investigate more in this field.

Silvestri et al.in[7]analyzed a healthcare dataset collected from hackers’news websites.This study adopted a machine learning model named Bidirectional Encoder Representations from Transformers(BERT) and eXtreme Gradient Boosting (XGBoost) and extracted information from the available documents on the web to determine the threats and vulnerabilities that impact the healthcare system.The results show an accuracy of 0.99.

On the other hand,the study[8]studied the Network traffic data to detect cybersecurity threats in the Internet of Things (IoT) Environment.The study used the N-BaIoT dataset and applied a supervised machine learning approach using Regularized Extreme Learning Machine (Regularized RELM)and Mafly optimization.The system achieved 98.93,98.95,and 98.94 precision,recall,and F-measure,respectively.

Shaukat et al.in [9] applied a supervised machine learning approach using Support Vector Machine (SVM),Deep Belief Network (DBN),Decision Tree (DT) models to evaluate the cyber threats detection on several different datasets Knowledge Discovery and Data Mining Tools Competition dataset (KDD CUP 99),Spambase,Twitter dataset,Enron,Neural Structured Learning–Knowledge Discovery in Databases (NSL-KDD),Defense Advanced Research Projects Agency(DARPA),and malware datasets.The results of this study show that using a specific learning technique for a particular cyber threat detection is not recommended.In addition,there is a need to investigate and analyze different learning models for cyber threat detection and build a benchmark dataset for that purpose.

Ke [10] studied some different machine learning methods to detect the threats in cybersecurity based.The study analyzed the NSL-KDD dataset using supervised machine learning models:Logistic Regression (LR) Multi-Layer Perceptron (MLP) and SVM and deep learning models: DBN and Stacked Non-Symmetric Deep Auto-Encode (SNSDAE).The results show that deep learning performs better than shallow learning to detect cybersecurity threats.

Moreover,the study[11]applied shallow supervised machine learning and deep learning models to examine its performance on three different cybersecurity issues:spam detection,intrusion detection,and malware analysis.The analyses used RF model and the Feedforward Deep Neural Network model on DGA datasets.The results show that there is a need for more investigation on using machine learning to deal with cybersecurity sensitive problems and the results still show some shortcomings that limit the applicability of machine learning models on cybersecurity issues.Nevertheless,the study expected that deep learning could have improved to deal with cybersecurity in the future.

Table 1 gives a summary of the most important literary works in the field of cyber threat detection using machine learning algorithms.

Table 1:Cyber threats detection literature works summary

In this work,an empirical investigation is conducted to detect cybersecurity threats in Twitter posts by using a multiclass classification machine learning approach to enrich the research in this field and explore the performance of using machine learning in the detection of cybersecurity threats.

3 Proposed Framework

The pipeline of the framework for analyzing data to detect cybersecurity threats is illustrated in Fig.1.The details for each stage in the proposed pipeline are explained in the next subsections.

Figure 1:The phases of the study framework pipeline

3.1 Data Acquisition

The data used in this study is secondary data published by[22].The data is collected from Twitter related to security issues.The dataset has 21368 data rows in four columns (id,text,target,type).The target (label) includes four classes,which are threat,business,irrelevant and unknown.Table 2 illustrates the statistics of each class.

Table 2:Tweets class statistical distribution

3.2 Data Preprocessing

Twitter data usually contains many undesired and irrelevant extra data that is considered noise which consumes more processing time and affects the analysis results [23].The data preprocessing stage removes these unwanted data and cleans it to be ready for further stages of analysis.To achieve this mission,some tasks should be accomplished.These tasks include Data cleaning,Tokenization,Removing stop words,and Stemming[24].

3.2.1 Data Cleaning and Filtering

In this task,the raw tweets passed through some steps to clean and remove unwanted and unrelated symbols and noisy non-word characters from the tweets such as the punctuations,special characters,and non-English words and symbols.These non-word characters affect the analysis process,consume a lot of processing resources,and affect the result accuracy[24].The output of this task is cleaned tweets that consist of cleaned words without any non-words characters.In addition,the filtering process is taking place to remove the duplicated and empty tweets that consume the processing resources without any benefits[25].

3.2.2 Tokenization

This task includes the process of breaking down the tweets into their basic units(tokens or words)by using the white space as a separator.Tokenization is a very important task where future stages of analysis id depend completely on its output [26].There are many tokenizing techniques used in literature,word tokenizer technique produced by Natural Language Tool Kit(NLTK)[27]is one of the famous tokenizers and it is used in this study.The expected output of this task is a list of lists(a list of tweets and each tweet is a list of words).

3.2.3 Removing Stopwords

Stopwords are the words that are used to connect sentences and it does not have meaning like determiners,prepositions,and coordinating conjunctions.Removing such words will save more of the analysis resources to focus on the important words instead.Each language has its stopwords.In this study,the stopwords of the English language are used,and the study uses the stopwords list that is published by NLTK[28].

3.2.4 Stemming

The stemming is a process to remove the suffixes and prefixes from the word to return it to the basic original form(stem).This process will help to find the true frequency of one word in one tweet or the whole dataset.In this study,snowball stemmer[29]is used to transform the tweet’s words into its basic stem.

3.3 Feature Extraction

The data produced in the preprocessing stage is still textual and it needs to be prepared for the machine learning processes.The machine is a digital platform that only understands numeric data[30].Machine learning algorithms accept numeric data and these data are called features.Features can be extracted from the textual data to be analyzed by the algorithm for learning.

In the feature extraction stage,valuable information will be extracted from the cleaned data as features.Many kinds of features can be extracted.This work utilizes two famous models for data representation used widely in the literature.The first one is called Bag of Words(BOW)[31]and the other model is TF-IDF[32],these models extract numeric features to be used in the machine learning algorithm.

The main idea behind the model BOW is the representation of all tweets in the dataset as a matrix where each row in that matrix represents one tweet while each column in that matrix represents one word of the whole tweets.After that,the frequency of appearance of each word(column)is calculated for each tweet(row)and held in the corresponding cell.Fig.2 illustrates the BOW representation for two different tweets selected randomly from the dataset.

From the above sample,d1 and d2 represent two different tweets and are considered as rows in the BOW matrix,while each word in those tweets is extracted and represented as a column in the BOW matrix.The frequency of word appearance in each tweet is held in the corresponding cell.For example,the word“best”appears in tweet d1 two times and the word“way”appears in both tweets d1 and d2 one time for each.The word frequency in each tweet is given by Eq.(1),which is called the Term Frequency(TF)equation as follows:

where w represents the tweet words,t is the tweet,andfw,t is the frequency of the word w occurring in tweet t[33].

Figure 2:The representation of two tweets using the BOW model

However,this way of representation has a drawback,suppose there is an irrelative word that has a high score of word frequency because it appears in one or two tweets with high frequency.Then TF representation will give that word high priority while it is not important for the general topic of the dataset.For that reason,the learning will be low in accuracy.To overcome this issue,the frequency of words is integrated with another concept to give a high score for the most important word in the dataset.This concept is IDF,which calculates the inverse fraction of all words in the dataset to give a correct explanation of the importance of each word related to the whole dataset.Eq.(2)calculates the IDF for the given word:

where t is a tweet,T represents all tweets in the dataset,|T|is the number of tweets,and|[t ∈T:w ∈t]|is the number of words w in all tweets[32].The idf(w,T)is integrated with TF to get the frequency of importance weight for each word in the whole dataset[34].Eq.(3)combines the two concepts.

3.4 Multiclass Classification

The results of the previous stage of feature extraction is a numeric data in form of TF-IDF values.This kind of data is suitable to be used to train and learn the machine.As mentioned earlier,the dataset that is used in this work contains tweets belonging to four different classes(threat,business,irrelevant and unknown).This arise a popular machine learning problem called(Multiclass classification)where the classification is usually applied to data with only two classes (binary classification) [35].For multiclass classification or multinomial classification,there are two main strategies followed to solve this problem:One-vs.-rest strategy which trains one classifier per one class where the samples of that class are considered as positive samples on the other side all the rest or other samples are considered as negative.The second strategy is one-vs.-one which trains one classifier for each pair of classes[36].Eq.(4)gives the number of class pairs to be trained:

where P is the total number of class pairs and N is equal to the total number of classes[37].

In this work,one-vs.-rest strategy is followed to classify the data using six different classification algorithms:SVM,Na?ve Bayes(NB),RF,LR,DT,and KNN.One-vs.-rest strategy is used widely in literature and it shows good results with classification algorithms such LR [38],SVM [37],and RF[39].Table 3 illustrates the parameter setting for each algorithm.

Table 3:Machine learning algorithm parameters settings

4 Experiment Results and Discussion

In this study,all the experiments are conducted using Python programming language and run using the Anaconda environment and its integrated editor of Jupiter Notebook.Windows operating system PC was used to run the all experiments with an Intel processor,Core i7-4770,speed of 3.40 GHz,and 8 GB RAM.

The experiments flow to start with preparing the data and preprocessing with a total number of 21368 tweets.Preprocessing stage outcomes out with 15519 tweets after filtering and removing the empty and duplicated tweets.Feature extraction is followed using the TF-IDF technique to extract the importance frequency of each feature in the dataset.A total number of 17805 features is obtained from the feature extraction process,which became in ready format to be fed into the machine learning algorithms.Before proceeding to the classification process,the whole dataset is divided into two parts:the training dataset and the testing dataset with 66%and 34%division,respectively.Each algorithm is trained using the training data then the prediction of the algorithm is tested using the testing data.

The result is evaluated by calculating the most popular evaluation metrics:precision,recall,F1-score,and accuracy.The confusion matrix is produced for each algorithm and visualized using clear readable diagrams.Fig.3 shows the confusion matrix and Table 4 shows the classification report for each algorithm.

Table 4:The classification results report

From the above evaluation results,the classification algorithms have achieved different scores of accuracy.Notably,the RF algorithm achieved the best results with the highest accuracy (0.67)among the other algorithms,while the DT achieved the lowest results and Na?ve Bayes achieved the lowest accuracy(0.44).In general,the results are quite low and there are some reasons behind this low accuracy.The first reason is related to the representation of unstructured textual data.The tweet text is short,unstructured,and has many issues that cause the low accuracy of the machine learning tasks[40].One of the issues in analyzing unstructured text is the high dimensionality of the resulting feature vectors[41].The second reason is related to the data representation technique TF-IDF,which resulted in a very sparse feature vector matrix.Most of the values of these sparse vectors are zeros and this affects negatively the accuracy of the analysis results[41,42].

Figure 3:The confusion matrix of the classification results:(a)RF result;(b)LR result;(c)DT result;(d)SVM result;(e)KNN result;(f)NB result

The classification result of the proposed study can be compared with the other methods of threat detection.Table 5 shows a brief comparison between some studies from the literature with the proposed study in the form of precision,recall,and F1-score.The chosen comparsion studies were selected based on the used data and the multiclass classification approach.From the comparison table,it is clear that the proposed study has achieved the best recall and F1 using the RF classification method among the other studies.The high F1-score indicates that the proposed method achieved good balanced predictions of true positive and true negative.As observed from the Table 5,the study[13]achieved high precision but low recall and F1-score,that is meaning the model can predict true positive accurately but it gives low accuracy with true negative prediction.This unbalanced results of precision and recall cause the low results of F1-score.

Table 5:Results comparison

5 Conclusion and Future Works

The detection of cyber threats is an important process and it is not an easy task.There are many works in the literature that used different approaches to overcome this task.The integrity between the several sciences opens a new analysis era to use other approaches in one field to solve issues in other fields.Machine learning is one of the promising approaches that achieve good results in several aspects of other fields.Resolving cybersecurity issues with machine learning still needs more investigations.In this study,a machine learning supervised multiclass classification approach is used to classify data from Twitter into several classes to detect cyber threads.By using several machine learning algorithms,this study achieved a detection accuracy of(67%).For future work,more investigation must be conducted to discover the other sides of cyber threat detection by using machine learning.In addition,the study suggests using different data representations for the feature extraction process such as Word2Vec to refine and obtain a high score of accuracy.Furthermore,the study suggests adding a new process to the framework pipeline which is the feature selection process to select the optimal set of features and ignore the unimportant features that affect the accuracy of the detection.

Acknowledgement:The authors extend their appreciation to the Deputyship for Research&Innovation,Ministry of Education in Saudi Arabia for funding this research work through Project Number MoE-IF-UJ-22-04100409-5.

Funding Statement:This research was funded by Deputyship for Research&Innovation,Ministry of Education in Saudi Arabia,Project Number MoE-IF-UJ-22-04100409-5.

Author Contributions:The authors confirm contribution to the paper as follows: study conception and design: A.Hussein,A.A.Almazroi;data collection: A.Hussein;analysis and interpretation of results:A.Hussein;draft manuscript preparation:A.Hussein,A.A.Almazroi.All authors reviewed the results and approved the final version of the manuscript.

Availability of Data and Materials:The data that support the findings of this study are openly available in“github”at http://doi.org/10.1109/BigData.2018.8622506,reference number[22].

Conflicts of Interest:The authors declare that they have no conflicts of interest to report regarding the present study.