Optimization of predictive performance of intrusion detection system using hybrid ensemble model for secure systems

Dataset description

The characteristics and features of datasets used in the evaluation of the proposed mode have been presented as follows.

NSL-KDD

In related work, most of the researchers used KDDCup99 for training their final systems. Although this dataset is good to train the classifiers, but it has some issues such as duplicate values and lack of real-world attacks scenarios. To develop a practically applicable robust intrusion detection system, this research work primarily used a newer dataset, NSL-KDD, to validate the performance of the proposed method, as it had real-world scenarios and millions of network records for NID evaluation. This dataset does not have redundant records, so any model trained on this dataset should not be biased towards repeated attack records. The NSL-KDD (Canadian Institute of Cybersecurity, 2022) consists of four sub-datasets that are KDD Test+, KDD Train, KDD Test-21, and _20 Percent. The dataset has records of various network attacks that an intrusion detection system must detect to avoid security problems. There are total of 43 features in this dataset for a single record. Out of 43 features, 41 are related to traffic input and the other two are labels and scores of traffic input. This dataset has a total of four classes for various attacks: probe, user to root (U2R), denial of services (DoS), and remote to local (R2L). A breakdown of number of records distributions is presented in the Table 2.

As mentioned, a DoS attack tries to shut down the system while the other three attacks are related to gaining access to system and sensitive information. This dataset suffers class imbalance problem where more than half of the records are normal. Distribution of U2R and R2L attacks is very low as compared to DoS attack. This property of the dataset resembles the real-life scenarios where DoS is the main type of attack that is a threat to the network and a smaller number of U2R and R2L attacks are seen in real environments. The input features in the dataset can be divided into four main categories that are: intrinsic, content, time-based, and host based.

UNSW-NB15

This dataset was created by the Australian Center for Cyber Security (Moustafa & Slay, 2015) having more than two million records, 48 features and nine different attack types. These features were extracted using newly developed algorithms such as Bro-IDS and Argus tools. The dataset was generated in the Cyber Range Lab where realistic state-of-the-art normal, and synthetic abnormal networks were established. The features are further categorized into five major groups which are flow features, basic features, content features, and time features along with additional generated features. The dataset contains nine types of different attacks: Fuzzers, Analysis, Backdoors, DoS, Exploits, Generic, Reconnaissance, Shellcode and Worm.

Table 3 presents the overall statistics of the UNSW-NB15 dataset.

CSE-CIC-IDS2018

The dataset was created by Communication Security Establishment (CSE) and Canadian Institute of Cybersecurity (CIC) in 2018. This work was aimed to develop user-profile based diversified instructional data for intrusion detection on networks. The data resembles the actual user behavior on the network. The dataset has seven different attack classes: DoS, DDoS, web attacks, botnet, heartbleed, brute-force, and infiltration. Due to the huge amount of dataset, only a part of data was used. CSV files, only from Friday-02-03-2018 and Friday-16-02-2018, were selected for the evaluation purposes. As the data is highly imbalanced, only those parts of the data were selected that could simulate real-world attack scenarios. The selected data has four classes: normal traffic, botnet, DoS-lowHTTP and DoS-Hulk, which are merged to represent a single class i.e., attack. After removing null values and duplicate records, 1,074,342 records were retained, out of which 290,089 records were malicious. Table 4 provides overall details of the dataset.

Preprocessing

As discussed above, raw dataset might not yield good results due to class imbalance, various data types and huge number of input features. Therefore, to develop robust machine learning models, input features were preprocessed to overcome the challenges. In the following section, an approach for dataset preprocessing is discussed in detail. The preprocessed datasets were later used in training machine learning algorithms.

Feature normalization

To develop a robust machine learning classifier, data normalization is required. It is the process of standardization of input data features to remove the biasness of the machine learning classifier. The data values are standardized by the maximum value in that data feature so that they lie in between standard values of (0–1). The normalization operation was applied by using Eq. (1). As a result of this process, all the numerical values were converted to a value range of 0 to 1.

(1) $$X^{\mathrm{\prime }}={\displaystyle \frac{x-\mu }{\sigma }}$$

In this equation, x is the original feature value, where X′ is the normalized value. $\sigma$ and $\mu$are standard deviation and mean respectively. Due to the normalization process, some features with high numerical values cannot affect the performance negatively. In addition, only 13 out of 41 features were used for final classification, in the case of NSL-KDD dataset. The detail of feature selection is provided in the next section.

Feature selection

There are total of 41 input features in the NSL-KDD dataset, while there are 45 and 73 input features in the UNSW-NB15 and CSE-CIC-IDS2018 datasets, respectively. Selected target labels might not be affected by some input features. Keeping this in view, only those features were retained which were required for classification purposes. For this task, RFE was utilized for feature selection (Guyon et al., 2002), which is a wrapper based backward feature selection method. In this technique a model is built using an entire set of input features. Then, the important score for each feature is calculated. In this recursive process, the least important features were removed by retraining models on various sets of features. Here, a subset of feature size is a tuning parameter to calculate specific number of features. When a subset gives optimal performance, that subset is used for predictors. The final optimal subset is used to train the final algorithm. In simple words, RFE performs a greedy optimization search to find the best performing feature subset for final model. Algorithm 1 describes the feature selection based on RF-RFE method.

All models cannot be paired with RFE method, because when number of features increase as compared to the number of samples, some models cannot be used with RFE. RFE with RF is used in this study for two reasons. First, RF does not tend to exclude the variables from prediction equation. An ensemble method usually has increased performance as compared to the individual models. RF, being an ensemble method, enforces the trees to have sub-optimal splits of features using random sample of input features. Second reason to include RF as base learner in RFE is because it has internal mechanism to measure the importance of features (Chen et al., 2018). Other than RF-RFE feature selection method, various other methods are widely utilized by the researchers. For example, in Gulla et al. (2020) researchers applied a combination of Gray Wolf Optimization (GWO) and Particle Swarm Optimization (PSO), however the results from this study are superior as compared to the Gulla et al. (2020) and a study in Roy et al. (2022) where k-best method was utilized. This research experimented with several methods for feature selection, however in this study only results of best performing method i.e., RF-RFE are discussed.

In addition, t-stochastic neighbor embedding (t-SNE) visualization was also used to visualize 41 features as shown in Figs. 2 and 3. T-distributed stochastic neighbor embedding (t-SNE) is a statistical data visualization method for high dimensional data. The method visualized the data points by giving each datapoint a location in a three-dimensional space. The method is based on stochastic neighbor embeddings. It is a nonlinear technique for data dimensionality reduction. It can be seen from the visualizations that the red dots overlap the blue and other dots representing different kinds of attacks in different regions. It becomes difficult for the learning algorithm to learn various distribution areas for the same type of attack traffic at the same time. The RF-RFE algorithm, applied to data, suggested only 13 features that were important for classification. In Fig. 4, t-SNE visualization for features selected by RF-RFE algorithm for final classification is shown. Features selected by RF-RFE help to differentiate the attack types and decreases the overlap between attacks.

Machine learning algorithms

Most ML algorithms need extracted features while deep learning, which is a subset of machine learning, does not require extracted features as it directly learns from the data. Instead of direct implementation of popular algorithms, in some cases, algorithms are merged to form ensemble models. In this research study, five individual ML models and three deep learning models along with proposed ensemble method were trained. Further details have been discussed in the upcoming sections.

Hyperparameters values

Hyperparameter values are necessary in machine learning approaches to train the proposed model. For deep learning models, basic and simple architectures were trained and tested to overcome overfitting as deep learning algorithms easily overfit on the training data. Architecture engineering in deep learning itself is a huge research domain. So, only training of basic variant of deep learning algorithms was considered and training the optimized deep learning algorithms for network intrusion detection was seen outside the scope of this work. Table 5 summarizes the key information about hyperparameters values used in the present research.

Ensemble method

The main idea behind the ensemble learning is to get the advantage from various classifiers by learning in a collective ensemble way. Each classifier has its own strengths and drawbacks for data classification. Some classifiers perform well on specific types of attacks while others may perform well on the rest of attack types. The key idea is to combine several weak classifiers by training multiple classification models and develop a strong classification model by utilizing a voting algorithm. In this way, weaknesses of the classification algorithms can be reduced to develop a strong classification model. The proposed ensemble model is based on RF, MLP and SVM classification models. These algorithms were trained collectively via a hybrid approach where RF-RFE helped to reduce the input features and later, a stacking ensemble method used the individual predictions to calculate the final class prediction using majority vote. The proposed method helped in minimizing the variance and increasing the predictive force of collective learners. The results showed that the performance gains were notable when utilizing the proposed ensemble method.

Figure 5 presents the basic idea of the proposed approach in this article. Algorithm 2 presents the steps in the proposed ensemble model training and testing.

Evaluation metrics

Several evaluation methods have been used to evaluate the performance of classifiers. Most of the metrics come from the information retrieval domain. Some of the popular metrics to evaluate the classifier are precision, recall, accuracy, F1-score, and mean absolute error (MAE). In this section, various evaluation metrices for classifiers’ evaluation are described. Many of these metrics can be calculated by using a confusion matrix. A typical confusion matrix is presented in Table 6. This confusion matrix forms the basis of various other metrics to evaluate the final classifier. The correct and incorrect predictions can be presented in a tabular form and a confusion matrix is generated. TN, TP, FP, FN are four values which are part of a confusion matrix.

Accuracy—It is the percentage of the correct predictions by the classifier. It can be calculated from the terms that are defined earlier in the confusion matrix section. It can be calculated by using a following formula:

(2) $$Accuracy={\displaystyle \frac{TP+TN}{TP+FP+TN+FN}}$$

Precision—It indicates the probability of a test instance that was positive and correctly identified as positive. It is given by:

(3) $$Precision={\displaystyle \frac{TP}{TP+FP}}$$

Recall—It is also called true positive rate. It gives us the indication of actual positive values which were identified correctly. It can be calculated by following formula:

(4) $$Sensitivity={\displaystyle \frac{TP}{TP+FN}}$$

F1-Score—It is the harmonic mean of sensitivity and precision. The final classifier can be evaluated using this evaluation metric. It can be calculated as:

(5) $$F1Score=2\ast {\displaystyle \frac{Precision\ast Recall}{Precision+Recall}}$$

Evaluation using NSL-KDD dataset

The detail of results for NSL-KDD with classification performance on original set of forty-one features is presented in Table 7. Results showed that for all features used, classifiers did not give higher percentages on the performance metrices. With all features utilized, the ensemble method showed better performance than the rest of the classifiers, yet the overall performance was not appreciable. For the next stage, classification performance based on feature selection method was investigated. In total, thirteen features were selected out of the original forty-one features in the dataset. The results showed that random forest classifier achieved an accuracy of 98.81% with 97.70% precision value. The recall and F1-score were obtained as 95.66% and 96.67% respectively. The lowest accuracy was achieved in the case of K-Nearest Neighbor algorithm with 94.72% accuracy score. This could be due to the reason that it is a very simple method and could not handle complex data. The overall details of the obtained results can be seen in the Table 8.

Experimental results showed that RF outperformed other algorithms in terms of accuracy and precision measures. The same trend can be seen in the literature where random forest performed better as compared to the other algorithms (Taher, Jisan & Rahman, 2019). There might be two reasons for these results. First, it works by voting mechanism where predictions from various decision trees are used to make final decision. The second reason could be the fact that by having multiple decision trees in the forest, only best features that contribute to the final prediction are selected for training the final classifier. Although MLP showed better results as a single classifier, when used in an ensemble model, considering its extensive training time, preference was given to KNN. Conclusively, ensemble of RF, SVM and MLP performed undoubtedly well with less training time and high percentages on performance metrices. Hence, efficient network intrusion detection systems can be developed by utilizing high performance machine learning algorithms. Confusion matrix for the NSL-KDD dataset is given in the Fig. 6 below.

Overall performance of machine learning algorithms has been shown through a bar graph in the Fig. 7, while performance of various machine learning methods on NSL-KDD dataset is shown in Table 8 shows result of each classifier with performance metrics mapped on the attacks included in NSL-KDD dataset. For each attack type, ensemble model outperformed individual algorithms. However, a slight variation was observed in recall and F1-score metrics for U2R and Probe attack. The trees in RF protect each other from their individual prediction errors. Although some trees could be wrong but many others will be correct at the same time. Therefore, the trees as a group can make the right prediction by helping each other in the prediction task. On other hand, the ensemble method is based on heterogeneous based classifiers which do not show similar behavior and the number of attacks in Probe and U2R are very limited. Due to this, RF performed better as compared to the method on U2R and Probe. Still, the proposed method was found much better as compared to the presented methods in the related literature.

Table 9 shows result of each classifier with performance metrices mapped on the attacks included in NSL-KDD dataset. For each attack type, ensemble model outperformed individual algorithms. However, a slight variation was observed in recall and F1-score metrics for U2R and Probe attack. The trees in RF protect each other from their individual prediction errors. Although some trees could be wrong, but many others will be correct at the same time. Therefore, the trees as a group can make the right prediction by helping each other in the prediction task. On other hand, the ensemble method is based on heterogeneous based classifiers which do not show similar behavior and the number of attacks in Probe and U2R are very limited. Due to this, RF performed better as compared to the proposed method on U2R and Probe. Still, the method was much better as compared to the presented methods in the related literature.

Figures 8 and 9 represent the ROC curve for the proposed ensemble model on the NSL-KDD dataset. It shows that the TPR rate is closer to 1, which is high, while the FPR is low as desired.

Table 9, Figs. 8 and 9 conclude that an IDS, using the proposed model, can effectively detect network intrusions.

Table 10 presents performance of the proposed system with some recent research works that utilized deep learning-based models for building intrusion detection systems using the NSL-KDD dataset. For a fair comparison, experiments with deep learning algorithms were carried out to demonstrate the superiority of the approach. It can be seen that the proposed approach consistently outperformed the deep learning algorithms in terms of performance measures, training time and testing time. One interesting observation was the lower model size of RNN trained model as compared to the proposed ML ensemble trained model in this study’s experiments.

Evaluation using the UNSW-NB15 dataset

To further evaluate the performance of the proposed approach, the preprocessed dataset was used. Tables 11 and 12 presents the experimental results of the proposed approach on this dataset.

As can be seen from Table 9, using all the features for classification, results in performance degradation. This is intuitive as some features are not always important for the classification. On the other hand, the results of this study suggested that ensemble performs well and shows the superior results, as seen in Table 9. The results show that RF performs better when comparing the individual algorithms with 87.9% accuracy. The lowest accuracy score was obtained in the case of KNN that was also the least performing algorithm in case of NSL-KDD. This reason for poor performance is discussed in the previous section. Reason for RF better performance is also already discussed in the previous section. Therefore, RF was chosen in the hybrid stacking ensemble method Table 12 shows the superiority of the proposed approach as compared to individual machine learning algorithms when using the proposed RF-RFE and the stacking-based ensemble method. The proposed method yields more than 98% accuracy which outperforms RF and other algorithms. Figure 10 shows the confusion matrix obtained for the whole dataset.

Overall performance of the algorithms and the ensemble approach is shown in the Fig. 11. The figure explains that the proposed ensemble method is outperforming all the individual algorithms in terms of classification metrics.

Figure 12 represent the ROC curve for the proposed ensemble model on UNSW-NB15 dataset. It shows that the TPR rate is closer to 1 which is high for majority classes, while the FPR is low as desired. Therefore, from the results presented in Table 12 and ROC figure, it may be concluded that an IDS, using the proposed approach, can effectively detect network intrusions. Figure 12 also show that ROC area under curve is nearly 1 for most classes which shows that the proposed method can effectively differentiate between these classes. However, in the case of class 1 (Backdoor) and class 8 (Worms) a sharp decline can be seen in the area under curve. This could be due to two reasons. Firstly, the dataset is highly imbalanced, and it has few samples as compared to other majority classes. Secondly, the dimensions in the overall dataset have been reduced using RF-RFE. This might have caused loss of important features that can increase performance for these minority classes. However, overall, the proposed method performs well as compared to the individual algorithms.

Table 13 compares performance of the proposed system with some recent research works that utilized the UNSW-NB15 dataset for intrusion detection. The researchers additionally compared the results with the three deep learning algorithms as well. Researchers have used deep learning, and machine learning-based models for building intrusion detection systems. This proposed method clearly outperforms previous works by considerable margins. Other than the model size of the RNN trained algorithm, this approach outperformed deep learning algorithms that were trained on the same machine. Mostly researchers only focused on binary classification therefore results in the study (Injadat et al., 2020) have better results as compared to the results of this study. However, information about attack classes is important in determining the countermeasures in case of intrusion. Therefore, superior results in binary classification might not be helpful in determining the intrusion countermeasures. In addition, mostly researchers do not provide the features that they used for classification and training and testing time requirements. As SMEs have limited resources, therefore, training and testing time might help them in selecting better choices for IDS development with limited resources and expertise.

Evaluation using CSE-CIC-IDS2018 dataset

As stated in “Dataset Description”, this dataset was only considered for binary classification (benign vs malicious) as the data is imbalanced with fewer records for specific classes. Additionally, the dataset is divided into a few files where each file is a record with the name of the day the record was generated. Combining the data into one file requires huge computational power and processing such a massive dataset was not possible due to resource limitation. Therefore, only binary classification is considered. Further details of the dataset are already discussed in “Dataset Description”.

Table 14 provides classification performance of the algorithms using original dataset features. In the case of this dataset, RF again performs better and KNN and MLP does not perform well. In comparison to individual algorithms, the ensemble method performs better. However, performance is still low as compared to the ensemble approach using RF-RFE selected features. In the case of original features RF performance is comparable to the proposed ensemble method in terms of precision and recall. This is also confirmed from the literature (de Souza, Westphall & Machado, 2022) where RF performs well. In Table 15 the performance of algorithms using features selected by RF-RFE is compared. It can be seen in the table that the acquired results are again 1% to 2% superior.

Figure 13 present the overall results of the machine learning algorithms and the ensemble method on RF-RFE features. Using RF as base learner in RFE helped in selecting useful features that help in final classification with few dimensions.

Figure 14 present the ROC curve for testing dataset. This shows a perfect area under curve of 1 in both cases. It implies that the proposed method was able to successfully differentiate between benign and malicious attacks on network.

To demonstrate the superiority of this approach, results were compared with the recent studies that utilized a similar dataset. As stated earlier, fewer researchers have investigated CSE-CIC-IDS2018 in the experiments, whereas NSL-KDD and KDD99 have been widely used. The approach was compared with a few recently published research articles and individual deep learning models. As seen in Table 16, the proposed method has performed much better than the deep learning approaches (Kim et al., 2020). Researchers have investigated newer methods for feature selection (Abdel-Basset et al., 2021) which performs better. However, it increases the computational overhead for training and testing of machine learning systems and SMEs cannot afford such expensive methods. When compared with other studies, the proposed approach is computationally friendly, and the results are also superior. One interesting thing found is the lower size of the deep learning trained models for CSE-CIC-IDS2018 when compared to the proposed model size. However, the performance of the system is slightly better than the individual algorithms with reduced training and testing time that is a requirement for real-time systems.

The proposed ensemble algorithm, based on hybrid feature selection, has outperformed individual algorithms on all three publicly available datasets. In case of NSL-KDD dataset, the method yielded more than 99% accuracy and F1-score that outperformed individual best performing model i.e., RF, while only using less than half of the original dataset features. At the same time, in comparison with the literature, the proposed method also outperformed the recent studies, as presented in Table 10. During investigation on UNSW-NB15 and CSE-CIC-IDS2018 datasets, same trends were observed where the proposed method outperformed the individual algorithms. The method yielded more than 98% F1-score and accuracy on UNSW-NB15 and more than 99% accuracy and F1-score on CSE-CIC-IDS2018 datasets. The detailed comparisons with recent studies are presented in Tables 14 and 16. The feature selection method helped in feature reduction while only selecting the best features that contributed towards the final prediction. This also helped in data dimensionality reduction and enhanced training and testing efficiency while generating state of the art intrusion detection results.

Implications

The proposed ensemble method excelled in performance metrices as compared to the individual machine learning algorithms. Moreover, in contrast to the deep learning approaches, that require large, labeled datasets, this simple ensemble method achieved the same performance without requiring high-end GPUs. This can be particularly helpful in the edge computing and internet of things (IoT) domain. As the edge devices do not have enough computation power, therefore methods based on deep learning may not work well in real-time in case of any intrusion. The first step to prevent attacks in networks is the timely detection of that attack. As discussed, in a resource constrained environment, deep learning methods might not work in real-time, so untimely detection of network intrusions can be disruptive in such scenarios. Moreover, computationally expensive deep learning models based NID systems are not a critical requirement in small and medium enterprises. SMEs can readily implement machine learning based systems and fulfil their need of combating cybercrimes.

Limitations and future work

To develop a machine learning based IDS, a comprehensive and representative of real-world attacks dataset is required. In this article, NSL-KDD is primarily used to perform extensive experiments. Although the results are superior on the dataset, a main challenge is to deploy the trained models on resource constrained devices such as edge devices and monitor the performance. In future, we would like to deploy and investigate the performance of such methods on edge devices.

Additionally, the UNW-NB15 dataset has been used to evaluate the performance of the proposed method. The dataset is imbalanced, and some classes have only a few handed samples (Backdoor and Worm). These minority classes affect the performance of the overall system that can be seen in the results and experimentation section. Therefore, CSE-CIC-IDS2018 dataset was further investigated to test the proposed approach. The dataset is comparatively new and contains millions of records. One problem with this dataset is the requirement for computational resources. Still, the method works well with this dataset and performs binary classification with more than 99% accuracy. Due to unavailability of computational resources, only binary classification was investigated on a subpart of the dataset. In future we would like to extend this work to the whole dataset in multiclass classification scenario.

In future, these datasets should be extended for more efficient development of network IDS. Moreover, different sectors should develop and share real-time network intrusion datasets to help researchers design sector specific intrusion detection systems. In the current study, machine learning has shown the ability to detect attacks in network. Therefore, we believe that with the development of newer intrusion datasets, a similar approach can be extended to work with the newer datasets.