Advertisement

Stacking ensemble and ECA-EfficientNetV2 convolutional neural networks on classification of multiple chest diseases including COVID-19

  • Mei-Ling Huang
    Correspondence
    Corresponding author: Mei-Ling Huang
    Affiliations
    Department of Industrial Engineering & Management, National Chin-Yi University of Technology, 57, Sec. 2, Zhongshan Rd., Taiping Dist., Taichung, 411030, Taiwan
    Search for articles by this author
  • Yu-Chieh Liao
    Affiliations
    Department of Industrial Engineering & Management, National Chin-Yi University of Technology, 57, Sec. 2, Zhongshan Rd., Taiping Dist., Taichung, 411030, Taiwan
    Search for articles by this author
Published:November 24, 2022DOI:https://doi.org/10.1016/j.acra.2022.11.027

      Abstract

      Background

      Early detection and treatment of COVID-19 patients is crucial. Convolutional neural networks have been proven to accurately extract features in medical images, which accelerates time required for testing and increases the effectiveness of COVID-19 diagnosis. This study proposes two classification models for multiple chest diseases including COVID-19.

      Methods

      The first is Stacking-ensemble model, which stacks six pre-trained models including EfficientNetV2-B0, EfficientNetV2-B1, EfficientNetV2-B2, EfficientNetV2-B3, EfficientNetV2-S and EfficientNetV2-M. The second model is self-designed model ECA-EfficientNetV2 based on ECA-Net and EfficientNetV2. Ten-fold cross validation was performed for each model on chest X-ray and CT images. One more dataset, COVID-CT dataset, was tested to verify the performance of the proposed Stacking-ensemble and ECA-EfficientNetV2 models.

      Results

      The best performance comes from the proposed ECA-EfficientNetV2 model with the highest Accuracy of 99.21%, Precision of 99.23%, Recall of 99.25%, F1-score of 99.20%, and AUC of 99.51% on chest X-ray dataset; the best performance comes from the proposed ECA-EfficientNetV2 model with the highest Accuracy of 99.81%, Precision of 99.80%, Recall of 99.80%, F1-score of 99.81%, and AUC of 99.87% on chest CT dataset. The differences for five metrics between Stacking-ensemble and ECA-EfficientNetV2 models are not significant.

      Conclusions

      Ensemble model achieves better performance than single pre-trained models. Compared with the SOTA, Stacking-ensemble and ECA-EfficientNetV2 models proposed in this study demonstrate promising performance on classification of multiple chest diseases including COVID-19.

      Keywords

      Introduction

      COVID-19 causes disease in humans and vertebrates, and is a zoonotic infectious disease. Some confirmed patients may have severe pneumonia and respiratory failure [

      C. Pneumonia, “COVID-19,” vol. 2019, no. December, 2019.

      ]. The most common tools for detecting lung infections are Chest X-ray (CXR) and Computer tomography (CT). Chest X-ray is a very common non-invasive radiological test that has been widely used to screen for a variety of lung diseases such as COVID-19, pneumonia, pulmonary effusion, lung cancer, and emphysema [
      • Liu H.
      • Wang L.
      • Nan Y.
      • Jin F.
      • Wang Q.
      • Pu J.
      SDFN: Segmentation-based deep fusion network for thoracic disease classification in chest X-ray images.
      ]. In clinical practice, chest X-ray images are often interpreted by radiologists, which is time-consuming and prone to errors in subjective assessments. A CT image is composed of a certain number of pixels with different grayscales arranged in a matrix. If the number of pixels is larger, the pixel value is smaller, and the image will be clearer. Although CT images provide very fine details, but it has more radiation than a chest X-ray image, and the equipment is relatively expensive [
      • Hou H.
      • et al.
      Artificial Intelligence Distinguishes COVID-19 from Community Acquired Pneumonia on Chest CT.
      ].
      In recent years, due to the rise of artificial intelligence, researchers have applied deep learning to detect COVID-19 by using chest X-ray images and CT images. Compared with traditional machine learning, deep learning can automatically extract features of images to reduce processing time [
      • Sharma N.
      • Sharma R.
      • Jindal N.
      Machine Learning and Deep Learning Applications-A Vision.
      ]. At present, there have been many researches applying convolutional neural network (CNN) on image recognition. This method has been proved to be a powerful image recognition technology and has been widely used for COVID-19 detection. Common CNNs include GoogLeNet, ResNet, Xception, DenseNet, MobileNet, and EfficientNet, etc. For example, Ieracitano et al. proposed a fuzzy logic-based convolutional neural network (CovNNet) and the classification accuracy is 81.00% in 2022 by using a total of 155 chest X-ray images [
      • Ieracitano C.
      • et al.
      A fuzzy-enhanced deep learning approach for early detection of Covid-19 pneumonia from portable chest X-ray images.
      ]. Khan et al. proposed two convolutional neural networks named DHL and DBHL in 2021, using a total of 6,448 chest X-ray images. The results showed that the accuracy of the binary classification of COVID-19 and normal was 98.53% [
      • Khan S.H.
      • et al.
      COVID-19 detection in chest X-ray images using deep boosted hybrid learning.
      ]. Loey et al. proposed a Bayesian-optimized convolutional neural network in 2022 using a total of 10,848 chest X-ray images. The results showed that the three-category accuracy of COVID-19, general pneumonia and normal was 96.00% [
      • Loey M.
      • El-Sappagh S.
      • Mirjalili S.
      Bayesian-based optimized deep learning model to detect COVID-19 patients using chest X-ray image data.
      ].
      Hu et al. proposed a two-stage detection method in 2021 using a total of 11,092 chest X-ray images [
      • Hu T.
      • Khishe M.
      • Mohammadi M.
      • Parvizi G.R.
      • Taher Karim S.H.
      • Rashid T.A.
      Realtime COVID-19 diagnosis from X-Ray images using deep CNN and extreme learning machines stabilized by chimp optimization algorithm.
      ]. The first stage is to train a CNN as a feature extractor, and the second stage uses extreme learning machines (ELMs) for real-time detection. In addition, the Chimp optimization algorithm (ChOA) is used to improve the results and increase the reliability of the network, and finally it is compared with the general CNN, Genetic algorithm optimized ELM (GA-ELM), Cuckoo search optimized ELM (CS-ELM) and Whale optimization algorithm optimized ELM (WOA-ELM). The results show that the proposed method has a binary classification accuracy of 99.11% for COVID-19 and non-COVID-19.
      Thaseen et al. applied ensemble learning by combining ResNet, FitNet, IRCNN, MobileNet, and EfficientNet in 2022, using a total of 13,808 chest X-ray images [
      • R S.
      • et al.
      An Efficient Hardware Architecture Based on An Ensemble of Deep Learning Models for COVID -19 Prediction.
      ]. The results showed that the three-category accuracy of COVID-19, general pneumonia and normal was 99.00%. A hybrid convolutional neural network (HCNN) combining CNN and RNN on classification of three-category accuracy of COVID-19, general pneumonia and normal was proposed with the classification accuracy of 98.20% [
      • Kumar M.
      • Shakya D.
      • Kurup V.
      • Suksatan W.
      COVID-19 prediction through X-ray images using transfer learning-based hybrid deep learning approach.
      ].
      Musallam et al. proposed a convolutional neural network called DeepChest in 2022, using a total of 7,512 chest X-ray images. The results showed that the three-category accuracy of COVID-19, general pneumonia and normal was 96.56% [
      • Musallam A.S.
      • Sherif A.S.
      • Hussein M.K.
      Efficient Framework for Detecting COVID-19 and Pneumonia from Chest X-Ray Using Deep Convolutional Network.
      ]. A convolutional neural network named DenResCov-19 composing of DenseNet121 and ResNet50 networks was proposed, and DenResCov-19 was combined with existing ResNet50, DenseNet121, VGG16 and InceptionV3 networks, using a total of 6,469 chest X-ray images. The results showed that the four-category AUC-ROC of COVID-19, general pneumonia, tuberculosis and normal was 99.60% [
      • Mamalakis M.
      • et al.
      DenResCov-19: A deep transfer learning network for robust automatic classification of COVID-19, pneumonia, and tuberculosis from X-rays.
      ].
      The literature on COVID-19 detection using CT images such as Garg et al. used a series of models such as EfficientNet, DenseNet, VGG, and ResNet with a total of 20 convolutional neural networks in 2022, using a total of 4,173 CT images. The results show that the binary classification accuracy of EfficientNet-B5 in detecting COVID-19 and non-COVID-19 is 98.45% [
      • Garg A.
      • Salehi S.
      • la Rocca M.
      • Garner R.
      • Duncan D.
      Efficient and visualizable convolutional neural networks for COVID-19 classification using Chest CT.
      ]. Lahsaini et al. used transfer learning on VGG16, VGG19, Xception, InceptionV2, ResNet, DenseNet121 and DenseNet201 in 2021, combined with GradCam, using a total of 4,986 CT images. The results show that in the binary classification of COVID-19 and non-COVID-19, DenseNet201+ GradCam achieves the best accuracy rate of 98.80% [
      • LAHSAINI I.
      • EL HABIB DAHO M.
      • CHIKH M.A.
      Deep transfer learning based classification model for covid-19 using chest CT-scans.
      ]. Rahimzadeh et al. used ResNet50V2 as the backbone network and compared the model with ResNet50V2 and Xception after adding a feature pyramid network (FPN), using a total of 63,849 CT images. The results show that ResNet50V2+ FPN has an accuracy of 98.49% in COVID-19 and normal binary classification [
      • Rahimzadeh M.
      • Attar A.
      • Sakhaei S.M.
      A fully automated deep learning-based network for detecting COVID-19 from a new and large lung CT scan dataset.
      ]. Qi et al. first used five models of UNet, LinkNet, R2UNet, Attention UNet and UNet++ to segment CT images, and then used pre-trained DenseNet121, InceptionV3 and ResNet50 for classification, using a total of Over 10,000 CT images. The results show that in the binary classification of COVID-19 and CAO, LinkNet performs best in lung segmentation with a Dice coefficient of 0.9830, while DenseNet121 with capsule network has a prediction accuracy of 97.10% [
      • Qi Q.
      • et al.
      Fully automatic pipeline of convolutional neural networks and capsule networks to distinguish COVID-19 from community-acquired pneumonia via CT images.
      ].
      Abdel-Basset et al. proposed a two-stage detection method in 2021. The first stage is to use the proposed GR-UNet to segment the area of lung infection, and then transfer learning is used as feature extraction; the second stage is to use the proposed GR-UNet to segment the lung infected area. The stage is to propose an infection prediction module that uses the infected location to make decisions about classification, using a total of 9,593 CT images. The results showed that the binary classification accuracy of COVID-19 and CAP was 96.80% [
      • Abdel-Basset M.
      • Hawash H.
      • Moustafa N.
      • Elkomy O.M.
      Two-Stage Deep Learning Framework for Discrimination between COVID-19 and Community-Acquired Pneumonia from Chest CT scans.
      ]. Ye et al. proposed a convolutional neural network named CIFD-Net in 2022, which can effectively handle the multi-region displacement problem through a new robust supervised learning, using a total of 45,167 CT images picture. The results showed that the binary classification accuracy of COVID-19 and non-COVID-19 was 91.19% [
      • Ye Q.
      • et al.
      Robust weakly supervised learning for COVID-19 recognition using multi-center CT images.
      ].
      Balaha et al. used transfer learning models including ResNet50, ResNet101, VGG16, VGG19, Xception, MobileNetV1, MobileNetV2, DenseNet121 and DenseNet169 in 2021, and added the harris hawks optimization (HHO) to optimize hyper-parameters, and finally use fast classification stage (FCS) and compact stacking stage (CSS) to stack the best models into one, using a total of 15,535 CT images. The results show that in the binary classification of COVID-19 and non-COVID-19, the weighted sum method (WSM) is used to obtain an accuracy of 99.33% [
      • Balaha H.M.
      • El-Gendy E.M.
      • Saafan M.M.
      CovH2SD: A COVID-19 detection approach based on Harris Hawks Optimization and stacked deep learning.
      ].
      Qi et al. proposed a detection method named DR-MIL in 2021, which first treats a 3D CT image of a patient as a bag and selects ten CT slices as initial instances. The deep features were then extracted from the pre-trained ResNet50 by fine-tuning and treated as a Deep Represented Instance Score (DRIS). The bag with DRIS was input to KNN to generate the final prediction, using a total of 241 patients CT images. The results showed that the binary classification accuracy of COVID-19 and CAP was 95.00% [
      • Qi S.
      • et al.
      DR-MIL: deep represented multiple instance learning distinguishes COVID-19 from community-acquired pneumonia in CT images.
      ].
      Some scholars used both chest X-ray images and CT images to detect COVID-19. Related studies such as Ahamed et al. fine-tune the pre-trained ResNet50V2 in 2021, using a total of 4,593 chest X-ray images and 3,000 CT images. The results showed that the four-category accuracy for detecting COVID-19, viral pneumonia, bacterial pneumonia and normal in chest X-ray images was 96.45%; the three-category accuracy for COVID-19, bacterial pneumonia and normal was 97.24%; COVID-19 and normal binary classification accuracy was 98.95%. The three-category accuracy for detecting COVID-19, CAP, and normal in CT images was 99.01%; the two-category accuracy for COVID-19 and normal was 99.99% [
      • Ahamed K.U.
      • et al.
      A deep learning approach using effective preprocessing techniques to detect COVID-19 from chest CT-scan and X-ray images.
      ]. Kumari et al. used four convolutional neural networks, InceptionV3, VGG16, Xception and ResNet50 in 2021, using a total of 2,000 chest X-ray images and 2,000 CT images. The results show that the binary classification accuracy of VGG16 in detecting COVID-19 and non-COVID-19 in chest X-ray images was 98.00%; the binary classification accuracy of Xception in detecting COVID-19 and non-COVID-19 in CT images was 83.00% [
      • Kumari S.
      • Ranjith E.
      • Gujjar A.
      • Narasimman S.
      • Aadil Sha Zeelani H.S.
      Comparative analysis of deep learning models for COVID-19 detection.
      ]. Ahsan et al. fine-tuned eight convolutional neural networks including VGG16, VGG19, ResNet15V2, InceptionResNetV2, ResNet50, DenseNet201, MobilenetV2, and NasNetMobile in 2020, using a total of 400 chest X-ray images and 400 CT images . The results show that the binary classification accuracy of NasNetMobile in detecting COVID-19 and non-COVID-19 in chest X-ray images was 100.00%; the binary classification accuracy of NasNetMobile in detecting COVID-19 and non-COVID-19 in CT images was 95.20% [

      M. M. Ahsan, K. D. Gupta, M. M. Islam, S. Sen, Md. L. Rahman, and M. S. Hossain, “Study of Different Deep Learning Approach with Explainable AI for Screening Patients with COVID-19 Symptoms: Using CT Scan and Chest X-ray Image Dataset,” 2020, doi: 10.3390/make2040027.

      ]. Jia et al. proposed a dynamic CNN modification method in 2021 and applied it to a fine-tuned ResNet, and finally compared the results with VGG16, InceptionV3, ResNet18, DenseNet121, MobileNetV3, and SqueezeNet, using a total of 7,592 chests X-ray images and 104,009 CT images. The results showed that the five-category accuracy of detecting COVID-19, viral pneumonia, bacterial pneumonia, tuberculosis and normal in chest X-ray images was 99.60%; in CT images, detecting COVID-19, non-COVID-19 and normal three-category accuracy was 99.30% [
      • Jia G.
      • Lam H.K.
      • Xu Y.
      Classification of COVID-19 chest X-Ray and CT images using a type of dynamic CNN modification method.
      ].
      Kassania et al. use eight models such as DenseNet, ResNet, MobileNet, InceptionV3, Xception, InceptionResNetV2, VGG and NASNet to extract features, and then extracted features were input to decision tree, random forest, XGBoost, AdaBoost, Bagging and LightGBM. A total of 137 chest X-ray images and 137 CT images were used. The results show that DenseNet121+ Bagging combines chest X-ray images and CT images to detect COVID-19 and normal with a binary classification accuracy of 99.00% [
      • Kassania S.H.
      • Kassanib P.H.
      • Wesolowskic M.J.
      • Schneidera K.A.
      • Detersa R.
      Automatic Detection of Coronavirus Disease (COVID-19) in X-ray and CT Images: A Machine Learning Based Approach.
      ].
      Gour and Jain proposed an integrated stacked CNN in 2022. After fine-tuning VGG19 and Xception and generating five sub-models, all sub-models were stacked using a softmax classifier with a total of 3,040 images Chest X-ray images and 4,645 CT images. The results showed that the three-category accuracy of detecting COVID-19, general pneumonia, and normal in chest X-ray images was 97.27%; and the two-category accuracy of detecting COVID-19 and normal in CT images was 98.30% [
      • Gour M.
      • Jain S.
      Automated COVID-19 detection from X-ray and CT images with stacked ensemble convolutional neural network.
      ]. Kamil et al. used fine-tuned VGG19 in 2021, using a total of 977 chest X-ray images and 23 CT images. The results showed a 99.00% accuracy of the binary classification between COVID-19 and normal [
      • Kamil M.Y.
      A deep learning framework to detect Covid-19 disease via chest X-ray and CT scan images.
      ].
      Saygılı applied Bag of Tree, K-ELM, KNN, and SVM with a total of 1,125 chest X-ray images and 3,228 CT images. The results showed that SVM has an accuracy of 85.96% in detecting COVID-19, general pneumonia and normal in chest X-ray images; K-ELM is accurate in detecting COVID-19 and non-COVID-19 in CT images. The accuracy was 98.88% [
      • Saygılı A.
      A new approach for computer-aided detection of coronavirus (COVID-19) from CT and X-ray images using machine learning methods.
      ].
      Existing pre-trained models are designed for general natural images and fine-tuned for classified image types, which is not specifically designed for COVID-19 detection. The general natural images are large and simple, while the images of COVID-19 have specific patterns and textures that differ significantly from natural images. Based on previous studies, we can find that the use of ensemble learning and the author's self-proposed convolutional neural network has good performance in detecting COVID-19 in chest X-ray images and CT images. Ensemble learning mainly combines multiple existing convolutional neural networks, which can not only reduce the probability of misjudgment by a single model, but also improve the classification accuracy in less time. Ensemble learning solves the current need to detect COVID-19 without designing a new model to get good detection performance in the most time-saving way.
      Challenging the possibility of improving accuracy on multiple chest disease diagnosis, this study proposes two classification models for multiple chest diseases including COVID-19. First, we obtained chest X-ray images and CT images from multiple public databases. Next, we select six pre-training models including EfficientNetV2-B0, EfficientNetV2-B1, EfficientNetV2-B2, EfficientNetV2-B3, EfficientNetV2-S and EfficientNetV2-M in the EfficientNetV2 series. The reason for choosing the above models is the training speed and argument efficiency of EfficientNetV2 is better than some previous networks [

      M. Tan and Q. v. Le, “EfficientNetV2: Smaller Models and Faster Training,” Apr. 2021, [Online]. Available: http://arxiv.org/abs/2104.00298

      ]. To the best of our knowledge, this is the first study ensembles the series of EfficientNetV2 models for COVID-19 detection. In addition, this study proposes a self-designed model ECA-EfficientNetV2 based on ECA-Net and EfficientNetV2.

      Materials and methods

      Figure 1 represents the architecture of this study.
      • Step 1 Data extraction: The chest X-ray and CT images are collected.
      • Step 2 Image pre-processing: The size of all selected images are equalized and saved in PNG. The datasets are split into training, validation, and test subsets.
      • Step 3 Pre-trained models: Six EfficientNetV2 models including EfficientNetV2-B0, EfficientNetV2-B1, EfficientNetV2-B2, EfficientNetV2-B3, EfficientNetV2-S, and EfficientNetV2-M are used.
      • Step 4 Proposed Ensemble-stacking model: The above six EfficientNetV2 models are stacked.
      • Step 5 Proposed model: ECA-EfficientNetV2 model.
      • Step 6 Performance evaluation: Accuracy, Precision, Recall, F1-score, and AUC are recorded for each model.
      • Step 7 Comparison with SOTA: the results of models are compared with the related studies.

      Data extraction

      Chest X-ray images and CT images are collected from five Kaggle datasets described as follows. The first Kaggle dataset is COVID-19 Radiography Dataset [

      “COVID-19 Radiography Database | Kaggle.” https://www.kaggle.com/tawsifurrahman/covid19-radiography-database (accessed Nov. 04, 2021).

      ], which contains 3,616 COVID-19, 1,345 viral pneumonia, 6,012 lung opacity and 10,192 normal for a total of 21,165 chest X-ray images (PNG). The second Kaggle dataset is Chest X-Ray Images (Pneumonia) [

      “Chest X-Ray Images (Pneumonia) | Kaggle.” https://www.kaggle.com/paultimothymooney/chest-xray-pneumonia (accessed Nov. 04, 2021).

      ], collected from pediatric patients aged 1 to 5 from Guangzhou Women and Children's Medical Center with 4,273 pneumonia and 1,583 normal chest X-ray images (JPEG). The third Kaggle dataset is Tuberculosis Chest X-ray Dataset [

      “Tuberculosis (TB) Chest X-ray Database | Kaggle.” https://www.kaggle.com/datasets/tawsifurrahman/tuberculosis-tb-chest-xray-dataset (accessed May 06, 2022).

      ], which contains 3,500 tuberculosis and 3,500 normal chest X-ray images (PNG). Large COVID-19 CT scan slice dataset [

      “Large COVID-19 CT scan slice dataset | Kaggle.” https://www.kaggle.com/maedemaftouni/large-covid19-ct-slice-dataset (accessed Nov. 09, 2021).

      ], containing 7,593 COVID-19, 2,618 chest abdominal pelvis and 6,893 normal CT images for a total of 17,104 CT images (PNG), is the forth Kaggle dataset used in this study. The fifth Kaggle dataset is COVID-19 & Normal& Pneumonia_CT_Images [

      “COVID-19&Normal&Pneumonia_CT_Images | Kaggle.” https://www.kaggle.com/datasets/anaselmasry/covid19normalpneumonia-ct-images (accessed May 08, 2022).

      ], containing 2,035 COVID-19, 3,309 pneumonia and 2,119 normal CT images for a total of 7,463 CT images (PNG).
      The COVID-19 X-ray images was formed based on the above datasets 1 to 3. The pneumonia images in the second dataset were split into groups of viral pneumonia and bacterial pneumonia. The total COVID-19 X-ray images contain five groups, which are 3,616 COVID-19, 2,780 viral pneumonia, 2,838 bacterial pneumonia, 3,500 Tuberculosis and 15,275 Normal for a total of 28,009 chest X-ray images (PNG).
      The COVID-19 CT images was formed based on the above datasets 4 and 5, which contains four groups including 9,628 COVID-19, 2,618 Chest Abdomen Pelvis (CAP), 3,309 pneumonia, and 9,012 Normal for a total number of 24,567 chest CT images.

      Image preprocessing

      To equalize the number of images for each group, 1,200 images are randomly selected for each group for both X-ray and CT datasets in this study. The number of images in training, validation and test subsets are 900, 100, and 200, respectively for X-ray dataset; The number of images in training, validation and test subsets are 4500, 500, and 400, respectively for CT dataset. Ten-fold cross validation was performed in this study. Tables 1 and 2 display the number of images for each group for COVID-19 X-ray dataset (Dataset X-ray) and COVID-19 CT dataset (Dataset CT) used in the study.
      Table 1COVID-19 X-ray dataset (Dataset X-ray)
      GroupTrainingValidationTest
      COVID-19900100200
      Bacterial Pneumonia900100200
      Tuberculosis900100200
      Viral Pneumonia900100200
      Normal900100200
      Total4,5005001,000
      Table 2COVID-19 CT dataset (Dataset CT)
      GroupTrainingValidationTest
      COVID-19900100200
      CAP900100200
      Pneumonia900100200
      Normal900100200
      Total3,600400800
      The original images collected from the above datasets differ in size and format. As the model requirement, all the images are pre-processed to in PNG with size of 224  ×  224. Fig. 2, Fig. 3 display the examples of images after pre-processed for chest X-ray (Dataset X-ray) and CT (Dataset CT) images, respectively.
      Fig 2
      Fig. 2Chest X-ray images after pre-processing

      Convolutional Neural Network

      Convolutional Neural Network (CNN) is a feedforward neural network, mainly composed of multiple convolution layers, pooling layers and fully connected layers. Compared with other neural networks, convolutional neural networks have better performances on image or speech recognitions. The goal of training a convolutional neural network is to find the most appropriate weights in the process of multiple forward and reverse iterations. Transfer learning is having previously trained model on a larger database that we can directly apply the architecture and weights of the pre-trained model to various studies to speed the efficiency of the training model. The Application module in Keras currently provides about 40 pre-trained models, all of which are trained on the ImageNet dataset. We choose the current newer and efficient EfficientNetV2, which includes six series of models including EfficientNetV2-B0, EfficientNetV2-B1, EfficientNetV2-B2, EfficientNetV2-B3, EfficientNetV2-S and EfficientNetV2-M. The reason for not choosing EfficientNetV2-L is that the model has a large amount of parameters and cannot be executed in our existing hardware resources.

      EfficientNetV2

      There are some problems with the previous EfficientNet series models, such as (1) when the size of the image is large, the training speed for EfficientNet-B3 to EfficientNet-B7 is very slow. (2) Training is slow when Depthwise convolutions is used. (3) It is not the best choice to enlarge each layer with the same magnification. Constantly increasing image size, which also leads to large memory consumption, which in turn affects training speed. Therefore, Tan & Le [

      M. Tan and Q. V. Le, “EfficientNetV2: Smaller Models and Faster Training,” 2021.

      ] proposed a new convolutional neural network EfficientNetV2 in 2021, which uses a non-uniform scaling strategy and can increase the number of deeper layers and has faster training speed and better parameter efficiency as compared to the previous models.
      In addition, Tan & Le [

      M. Tan and Q. V. Le, “EfficientNetV2: Smaller Models and Faster Training,” 2021.

      ] also used EfficientNet-B4 to do some experiments, and found that replacing the MBConv module with the Fused-MBConv module in the early stage can greatly improve the training speed. However, if each layer is replaced with the Fused-MBConv module, the amount of parameters and FLOPs will be significantly increased, and the training speed will be relatively reduced. Therefore, Neural Architecture Search (NAS) was used to find the best combination of MBConv and Fused-MBConv. Figure 4 is the architecture of EfficientNetV2.

      ECA-Net

      Channel attention has greatly improved the performance of convolutional neural networks, and most scholars are currently working on developing more complex attention modules. The most representative method such as Hu et al. [

      J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu, “Squeeze-and-Excitation Networks,” Sep. 2017, [Online]. Available: http://arxiv.org/abs/1709.01507

      ] used Squeeze-and-excitation (SE) module and proposed SE-Net in 2017. SE-Net first uses a global average pooling layer for each channel, and then uses two non-linear fully connected layers and a sigma function to generate the weights for each channel. Although the SE module is widely used in some researches on the current channel attention module, it has been proved that dimensionality reduction will affect both the prediction performance for channel attention, and the efficiency of obtaining the weights between all channels.
      Wang et al. [

      Q. Wang, B. Wu, P. Zhu, P. Li, W. Zuo, and Q. Hu, “ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks,” Oct. 2019, [Online]. Available: http://arxiv.org/abs/1910.03151

      ] uses a lightweight and efficient channel attention (ECA) module, which only adds a small number of parameters, but can achieve significant performance gains. The ECA module does not use dimensionality reduction and operates through a one-dimensional convolution of size k. According to the experimental results of ECA-Net on ImageNet-1K and MS COCO, ECA-Net has lower model complexity than the state-of-the-art methods. In addition, ECA-Net has better efficiency no matter in image classification, image segmentation or object detection.

      The proposed stacking-ensemble model

      Ensemble learning is a type of supervised learning that has been widely used in the fields of statistics, machine learning and deep learning. Compared with a single learning algorithm, the purpose of ensemble learning is to combine multiple algorithms or models to form a model with better predictive performance. Using an ensemble approach will yield better results when there are significant model-to-model differences [
      • Opitz D.
      • Maclin R.
      Popular ensemble methods: An empirical study.
      ]. [

      M. A. Ganaie and M Hu. “Ensemble deep learning: A review,” arXiv preprint arXiv:2104.02395 (2021).

      ] reviews on ensemble deep leaning. Details of ensemble classifiers with improved overfitting have been investigated by several studies, for instance [
      • Sollich P.
      • Krogh A.
      Learning with ensembles: how over-fitting can be useful.
      ,
      • Pourtaheri Z.K.
      • Zahiri S.H.
      Ensemble classifiers with improved overfitting.
      ]. In recent years, due to the continuous improvement of computing power of computers, large-scale integrated models can be trained within a reasonable time, and have been applied on medical image recognition, face recognition, emotion recognition and financial decision-making, etc. Successful applications of ensemble classifiers could be seen in [
      • Rajaraman S.
      • Kim I.
      • Antani S.K.
      Detection and visualization of abnormality in chest radiographs using modality-specific convolutional neural network ensembles.
      ,
      • Devnath L.
      • Fan Z.
      • Luo S.
      • Summons P.
      • Wang D.
      Detection and Visualisation of Pneumoconiosis Using an Ensemble of Multi-Dimensional Deep Features Learned from Chest X-rays.
      ,
      • Rajaraman S.
      • Cemir S.
      • Xue Z
      • Alderson P.
      • Thoma G.
      • Antani S.
      A Novel Stacked Model Ensemble for Improved TB Detection in Chest Radiographs.
      ,
      • Devnath L.
      • Luo S.
      • Summons P.
      • Wang D.
      • Shaukat K.
      • Hameed I.A.
      • Alrayes F.S.
      Deep Ensemble Learning for the Automatic Detection of Pneumoconiosis in Coal Worker's Chest X-ray Radiography.
      ].
      The main methods of ensemble learning can be divided into three categories: bagging, boosting and stacking. The main function of stacking is to combine multiple algorithms to make predictions, that is, the result integration of the voting method or the average method. Ensemble stacking is a powerful CNN ensemble method thanks to its unique meta-learning algorithm. Meta-learning algorithms work by taking the probabilities of the input from each sub-model and determining which model performs best in extracting features. The learning algorithms directly extend the learning of each sub-model and combine the best predictions from each model. If each model is unique, then each model learns differently. Stacking achieves better results than either trained model and is used to evaluate the error rate of bagging. This study chooses stacking as our approach for ensemble learning. The stacking-ensemble model developed in this study selects six pre-trained models including EfficientNetV2-B0, EfficientNetV2-B1, EfficientNetV2-B2, EfficientNetV2-B3, EfficientNetV2-S and EfficientNetV2-M in the EfficientNetV2 series. Figure 5 presents the architecture of the proposed stacking-ensemble model in this study.

      The proposed ECA-EfficientNetV2 model

      In addition to the proposed stacking-ensemble model, challenging the possibility of improving accuracy on multiple chest disease diagnosis, this study proposes a self-designed model ECA-EfficientNetV2 based on ECA-Net and EfficientNetV2. Motivation of choosing the above two models as basis are described as follows:
      • 1
        The advantages of EfficientNetV2 includes: (1) The network is better than some previous networks in terms of training speed and number of parameters; (2) It proposes improved incremental learning methods that dynamically adjust regularization (e.g., dropout, data augmentation, and mixup) according to the size of the training images; (3) Progressive learning is used to perform well on pre-trained ImageNet, CIFAR, Cars, and Flowers datasets.
      • 2
        The major advantages of ECA-Net are: (1) ECA is a lightweight attention module that only contains k parameters (k≤9), which can be used to improve the performance of large convolutional neural networks; (2) The ECA module uses a dimensionality-free local cross-channel interaction method that adaptively selects suitable adjacent channels to compute attention; (3) The features extracted by the SE module between different channels are relatively similar, while the features extracted by the ECA module from different channels are different, which indicates that the features extracted by the ECA module are better for classification.
      Based on the above two points, the proposed ECA-EfficientNetV2 are designed and introduced as follows:
      • 1
        ECA-EfficientNetV2 uses the dilated convolution module as the first two layers. The advantage is that while the kernel size is increased, the parameters or calculation amount of the original model can be maintained. Each module contains two dilated convolutional layers and an activation function SELU, where the dilation rate of the first layer is set to 2, and the dilation rate of the second layer is set to 3.
      • 2
        To reduce both the number of parameters and the computation complexity, ECA-EfficientNetV2 replaces SE module with ECA module in MBConv and Fused-MBConv convolution modules, and renames as MBEConv and Fused-MBEConv.
      • 3
        Zero-padding is added to the convolutional layers in both the MBEConv and Fused-MBEConv modules to prevent the input image be affected by the kernel size. In addition, the original activation function SELU is changed to ReLU to overcome the problem of vanishing gradient.
      • 4
        After MBEConv and Fused-MBEConv, two general convolution modules are added. The internal parameters include Zero-padding, Stride and SELU. In addition, we add a batch normalization after each convolutional layer, which makes training easier and more stable, and improves the performance of the neural network.
      • 5
        Use the global average pooling layer to improve the problem of a large number of parameters that occurs in the fully connected layer. After that, add a dropout layer before the classification layer to generate multiple results by continuously updating the weights, and finally remove the outliers to avoid the problem of overfitting.
      The number of parameters for ECA-EfficientNetV2 is 5,706,965, which is much less than the number of parameters (117,753,253) used in EfficientNetV2-L. Figure 6 presents the architecture of the proposed Fused-MBEConv layer and MBEConv layers, and Figure 7 shows the architecture of the proposed ECA-EfficientNetV2.
      Fig 6
      Fig. 6The architecture of the proposed Fused-MBEConv layer and MBEConv layers.
      (H, W, C denotes network height, width and channel respectively.)

      Model evaluation measures

      The performance indices include accuracy, precision, recall, F1-Score and the area under the curve (AUC). The mathematical formulas of these indicators are shown in Eqs. (1)-(5), respectively. Accuracy is the ratio of the number of correctly classified samples to the total number of samples; Precision is the ratio of the number of true positives to the total number of elements labelled to the positive class; Recall presents the number of true positives divided by the total number of true positives and false negatives; F1-Score is a measure of precision and recall. The higher the F1-score, the better the classification performance of the model; AUC denotes the measure at distinguishing between the positive and negative classes. Higher the AUC, better the model.
      Accuracyc=(TPc+TNc)(TPc+FPc+FNc+TNc)
      (1)


      Precisionc=TPc(TPc+FPc)
      (2)


      Recallc=TPc(TPc+FNc)
      (3)


      F1score=2nc=1nRecallc×PrecisioncRecallc+Precisionc
      (4)


      AUC=cnranknTPc×(1+TPc)2TPc×TNc
      (5)


      where c is the number of classes; TP (true positive) represents the number of positive categories that are correctly classified as positive, FP (false positive) represents the number of negative categories that are incorrectly classified as positive, TN (true negative) refers to the number of negative categories that are correctly classified as negative, and FN (false negative) refers to the number of positive categories that are incorrectly classified as negative.

      Results

      The equipment used in the experiment is an Intel(R) Core(TM) i9-10900F 2.81 GHz CPU, NVIDIA GeForce RTX 3070 8G GPU. The whole experiment process is performed using Python 3.8 [Python Software Foundation, Fredericksburg, Virginia, USA], which contains Keras 2.6 and Tensorflow 2.6. Ten-fold cross validation is used to evaluate the performance of each model under batch size of 16; epochs count of 30; optimizer of Adam, learning rate of 1e-5 and dropout of 0.4.

      Performance results on dataset X-ray

      On dataset 1, Table 3, Table 4, Table 5, Table 6 report the results of Accuracies, Precisions, Recalls, F1-Scores, and AUCs in the test sets, and the average and standard deviation for EfficientNetV2-B0, EfficientNetV2-B1, EfficientNetV2-B2, EfficientNetV2-B3, EfficientNetV2-S, EfficientNetV2-M, the proposed stacking-ensemble model, and the proposed ECA-EfficientNetV2 models. The best performance comes from the proposed ECA-EfficientNetV2 model with the highest Accuracy of 99.21%, Precision of 99.23%, Recall of 99.25%, F1-score of 99.20%, and AUC of 99.51%. Fig. 8, Fig. 9 show the examples of training accuracy and loss, confusion matrix and ROC curve for each model, respectively.
      Table 3Performance of EfficientNetV2-B0 and EfficientNetV2-B1 models
      FoldEfficientNetV2-B0EfficientNetV2-B1
      AccuracyPrecisionRecallF1-scoreAUCAccuracyPrecisionRecallF1-scoreAUC
      182.10%82.33%82.14%82.13%88.81%83.10%83.23%83.14%83.11%89.44%
      285.20%84.99%85.24%85.06%90.75%85.90%85.81%85.93%85.85%91.19%
      388.50%88.62%88.53%88.57%92.81%86.70%86.69%86.72%86.59%91.69%
      485.50%85.73%85.51%85.49%90.94%82.30%82.26%82.34%82.25%88.94%
      585.90%85.93%85.91%85.96%91.19%84.30%84.47%84.32%84.34%90.19%
      684.40%84.46%84.42%84.43%90.25%85.20%85.15%85.23%85.16%90.75%
      785.60%85.62%85.64%85.56%91.00%86.50%86.40%86.54%86.42%91.56%
      884.40%84.59%84.46%84.34%90.25%85.82%85.94%85.81%85.84%91.13%
      984.50%84.80%84.51%84.54%90.31%88.90%89.01%88.92%88.93%93.06%
      1086.20%86.42%86.23%86.22%91.38%87.60%87.79%87.64%87.65%92.25%
      Average85.23±

      1.63%
      85.35±

      1.61%
      85.26±

      1.63%
      85.23±

      1.65%
      90.77±

      1.02%
      85.63±

      2.00%
      85.68±

      2.02%
      85.66±

      2.00%
      85.61±

      2.01%
      91.02±

      1.25%
      Table 4Performance of EfficientNetV2-B2 and EfficientNetV2-B3 models
      FoldEfficientNetV2-B2EfficientNetV2-B3
      AccuracyPrecisionRecallF1-scoreAUCAccuracyPrecisionRecallF1-scoreAUC
      179.90%79.85%79.91%79.76%87.44%85.40%85.72%85.43%85.37%90.88%
      284.40%84.63%84.42%84.30%90.25%86.30%86.19%86.32%86.22%91.44%
      386.60%87.08%86.61%86.43%91.69%87.00%87.27%87.42%86.97%91.88%
      481.70%81.71%81.74%81.63%88.56%84.20%84.06%84.23%84.11%90.12%
      583.70%83.80%83.72%83.74%89.81%85.70%85.84%85.71%85.69%91.06%
      680.60%80.98%80.64%80.59%87.88%84.60%84.94%84.62%84.61%90.38%
      783.20%83.14%83.23%83.15%89.50%86.40%86.67%86.42%86.38%91.50%
      887.30%87.36%87.34%87.27%92.06%85.20%85.49%85.21%85.27%90.75%
      980.60%80.89%80.64%80.65%87.88%87.10%87.22%87.14%87.12%91.94%
      1085.10%85.31%85.14%85.15%90.69%85.90%86.26%85.94%85.92%91.19%
      Average83.31±

      2.59%
      83.48±

      2.63%
      83.34±

      2.59%
      83.27±

      2.58%
      89.58±

      1.62%
      85.78±

      0.96%
      85.97±

      0.99%
      85.84±

      1.02%
      85.77±

      0.96%
      91.11±

      0.60%
      Table 5Performance of EfficientNetV2-S and EfficientNetV2-M models
      FoldEfficientNetV2-SEfficientNetV2-M
      AccuracyPrecisionRecallF1-scoreAUCAccuracyPrecisionRecallF1-scoreAUC
      186.90%86.82%86.94%86.81%91.81%87.30%87.29%87.34%87.21%92.06%
      288.30%88.31%88.34%88.27%92.69%88.70%88.74%88.71%88.68%92.64%
      387.70%88.01%87.92%87.95%92.44%89.00%89.04%89.27%88.97%93.13%
      487.00%87.37%87.42%87.03%91.88%87.70%87.65%87.72%87.67%92.31%
      587.50%87.55%87.52%87.48%92.19%89.40%89.43%89.41%89.44%93.38%
      687.00%86.93%87.12%86.91%91.88%89.70%89.71%89.73%89.65%93.56%
      787.10%87.13%87.21%87.05%91.94%89.00%88.90%89.29%88.94%93.13%
      888.00%88.03%88.28%87.98%92.50%89.70%89.72%89.74%89.69%93.56%
      987.10%87.13%87.14%87.11%91.94%86.70%86.98%86.74%86.76%91.69%
      1087.50%87.69%87.53%87.56%92.19%89.30%89.33%89.31%89.34%93.31%
      Average87.41±

      0.47%
      87.50±

      0.51%
      87.54±

      0.49%
      87.42±

      0.51%
      92.15±

      0.31%
      88.65±

      1.05%
      88.68±

      1.01%
      88.73±

      1.07%
      88.64±

      1.05%
      92.88±

      0.66%
      Table 6Performance of Stacking-ensemble and ECA-EfficientNetV2 models
      FoldProposed Stacking-ensemble modelProposed ECA-EfficientNetV2
      AccuracyPrecisionRecallF1-scoreAUCAccuracyPrecisionRecallF1-scoreAUC
      198.40%98.42%98.47%98.39%99.00%98.80%98.84%98.81%98.79%99.25%
      298.70%98.71%98.74%98.72%99.19%99.60%99.64%99.61%99.59%99.75%
      398.90%98.91%98.93%98.94%99.31%98.80%98.82%98.84%98.78%99.25%
      498.50%95.54%98.51%98.53%99.06%99.50%99.51%99.54%99.49%99.69%
      599.00%99.01%99.04%98.99%99.38%99.10%99.12%99.11%99.09%99.44%
      699.00%99.02%99.05%98.99%99.38%99.40%99.41%99.44%99.37%99.62%
      799.00%99.05%99.03%99.01%99.38%99.50%99.48%99.54%99.53%99.69%
      898.30%98.35%98.31%98.29%98.94%99.00%99.04%99.07%98.99%99.38%
      998.60%98.64%98.67%98.62%99.13%99.00%99.02%99.07%98.97%99.38%
      1098.70%98.73%98.71%98.75%99.19%99.40%99.41%99.44%99.38%99.62%
      Average98.71±

      0.26%
      98.44±

      1.05%
      98.75±

      0.26%
      98.72±

      0.26%
      99.20±

      0.16%
      99.21±

      0.30%
      99.23±

      0.30%
      99.25±

      0.30%
      99.20±

      0.31%
      99.51±

      0.19%
      Fig 8
      Fig. 8Examples of training accuracy and loss from each model
      Fig 9
      Fig. 9Examples of confusion matrix and ROC curve from each model

      Performance results on dataset CT

      On dataset 2, Table 7, Table 8, Table 9, Table 10 report the results of Accuracies, Precisions, Recalls, F1-Scores, and AUCs in the test sets, and the average and standard deviation for EfficientNetV2-B0, EfficientNetV2-B1, EfficientNetV2-B2, EfficientNetV2-B3, EfficientNetV2-S, EfficientNetV2-M, the proposed stacking-ensemble model, and the proposed ECA-EfficientNetV2 models. The best performance comes from the proposed ECA-EfficientNetV2 model with the highest Accuracy of 99.81%, Precision of 99.80%, Recall of 99.80%, F1-score of 99.81%, and AUC of 99.87%. Fig. 10, Fig. 11 show the examples of training accuracy and loss, confusion matrix and ROC curve for each model, respectively.
      Table 7Performance of EfficientNetV2-B0 and EfficientNetV2-B1 models
      FoldEfficientNetV2-B0EfficientNetV2-B1
      AccuracyPrecisionRecallF1-scoreAUCAccuracyPrecisionRecallF1-scoreAUC
      195.13%95.22%95.15%95.10%96.75%93.63%93.65%93.62%93.59%95.75%
      293.87%93.88%93.85%93.86%95.92%96.25%96.41%96.25%96.24%97.50%
      395.63%95.60%95.62%95.59%97.08%95.13%95.32%95.15%95.07%96.75%
      495.13%95.09%95.15%95.08%96.75%93.25%93.20%93.25%93.17%95.50%
      595.63%95.78%95.62%95.60%97.08%93.00%93.03%93.17%92.88%95.33%
      694.25%94.29%94.27%94.23%96.17%93.25%93.37%93.27%93.28%95.50%
      793.25%93.43%93.28%93.13%95.50%94.13%94.35%94.17%94.04%96.08%
      894.37%94.39%94.35%94.36%96.25%90.87%90.85%90.88%90.77%93.92%
      993.50%93.51%93.54%93.47%95.67%95.13%95.11%95.15%95.17%96.75%
      1094.63%94.60%94.56%94.61%96.42%91.37%91.27%91.75%91.27%94.25%
      Average94.54±

      0.84%
      94.58±

      0.83%
      94.54±

      0.83%
      94.50±

      0.85%
      96.36±

      0.56%
      93.60±

      1.67%
      93.66±

      1.74%
      93.67±

      1.61%
      93.55±

      1.70%
      95.73±

      1.11%
      Table 8Performance of EfficientNetV2-B2 and EfficientNetV2-B3 models
      FoldEfficientNetV2-B2EfficientNetV2-B3
      AccuracyPrecisionRecallF1-scoreAUCAccuracyPrecisionRecallF1-scoreAUC
      194.87%98.88%94.85%94.83%96.58%94.37%94.54%94.38%94.36%96.25%
      297.00%96.99%97.41%96.87%98.00%95.63%95.65%95.62%95.61%97.08%
      394.87%94.92%94.88%94.84%96.58%94.87%95.02%94.85%94.83%96.58%
      495.00%94.97%95.02%94.94%96.67%95.13%95.10%95.12%95.08%96.75%
      595.00%94.99%95.17%94.97%96.67%96.37%96.39%96.38%96.36%97.58%
      692.87%92.95%92.88%92.85%95.25%95.75%95.74%95.75%95.73%97.17%
      792.13%92.19%92.15%92.03%94.75%95.00%95.21%95.14%94.97%96.67%
      894.63%94.58%94.65%94.59%96.42%95.50%95.54%95.51%95.48%97.00%
      995.25%95.24%95.19%95.24%96.83%94.37%94.36%94.35%94.76%96.25%
      1092.50%92.45%92.47%92.38%95.00%94.00%93.98%94.37%93.97%96.00%
      Average94.41±

      1.48%
      94.82±

      2.04%
      94.47±

      1.57%
      94.35±

      1.49%
      96.28±

      0.99%
      95.10±

      0.73%
      95.15±

      0.72%
      95.15±

      0.68%
      95.12±

      0.70%
      96.73±

      0.49%
      Table 9Performance of EfficientNetV2-S and EfficientNetV2-M models
      FoldEfficientNetV2-SEfficientNetV2-M
      AccuracyPrecisionRecallF1-scoreAUCAccuracyPrecisionRecallF1-scoreAUC
      196.37%96.36%96.35%96.63%97.58%95.25%95.22%95.27%95.22%96.83%
      296.63%96.71%96.62%96.65%97.75%96.13%96.14%96.15%96.10%97.42%
      397.38%97.40%97.37%97.39%98.25%95.50%95.54%95.51%95.47%97.00%
      496.63%96.62%96.65%96.33%97.75%95.13%95.12%95.15%95.10%96.75%
      596.50%96.47%96.54%96.48%97.67%96.75%96.76%96.73%96.78%97.83%
      695.75%95.74%95.77%95.78%97.17%96.00%96.05%96.04%95.99%97.33%
      796.50%96.58%96.54%96.51%97.67%95.25%95.28%95.26%95.24%96.83%
      896.88%96.86%96.87%96.89%97.92%96.50%96.51%95.54%96.49%97.67%
      995.63%95.67%95.65%95.62%97.08%95.37%95.33%95.35%95.34%96.92%
      1095.13%95.09%95.12%95.10%96.75%94.75%94.70%94.77%94.72%96.50%
      Average96.34±

      0.66%
      96.35±

      0.67%
      96.35±

      0.66%
      96.34±

      0.67%
      97.56±

      0.44%
      95.66±

      0.65%
      95.67±

      0.67%
      95.58±

      0.57%
      95.65±

      0.66%
      97.11±

      0.43%
      Table 10Performance of Stacking-ensemble and ECA-EfficientNetV2 models
      FoldStacking-ensemble modelECA-EfficientNetV2
      AccuracyPrecisionRecallF1-scoreAUCAccuracyPrecisionRecallF1-scoreAUC
      199.00%99.01%99.24%99.89%99.33%99.38%99.34%99.35%99.37%99.58%
      298.62%98.67%98.63%98.54%99.08%99.88%99.87%99.85%99.89%99.92%
      399.38%99.46%99.74%99.25%99.58%99.75%99.72%99.71%99.77%99.83%
      498.75%98.74%98.81%98.93%99.17%99.25%99.27%99.24%99.28%99.50%
      598.75%98.78%98.87%98.79%99.17%99.88%99.85%99.87%99.84%99.92%
      699.88%99.87%99.85%99.89%99.92%99.94%99.92%99.96%99.95%99.98%
      798.25%98.33%98.41%98.24%98.83%100.00%100.00%100.00%100.00%100.00%
      898.75%98.77%98.75%98.72%99.17%100.00%100.00%100.00%100.00%100.00%
      998.88%98.87%98.85%98.86%99.25%100.00%100.00%100.00%100.00%100.00%
      1098.13%98.16%98.15%98.12%98.75%100.00%100.00%100.00%100.00%100.00%
      Average98.84±

      0.51%
      98.87±

      0.50%
      98.93±

      0.54%
      98.92±

      0.61%
      99.23±

      0.34%
      99.81±

      0.27%
      99.80±

      0.27%
      99.80±

      0.28%
      99.81±

      0.27%
      99.87±

      0.18%
      Fig 10
      Fig. 10Examples of training accuracy and loss from each model
      Fig 11
      Fig. 11Examples of confusion matrix and ROC curve from each model

      Discussion

      Dataset X-ray

      The performances of 5 metrics among EfficientNetV2-B0, EfficientNetV2-B1, EfficientNetV2-B2, EfficientNetV2-B3, EfficientNetV2-S, EfficientNetV2-M, the proposed stacking-ensemble model, and the proposed ECA-EfficientNetV2 models for dataset X-ray were compared in Table 11. Two accuracies are above 98%, which come from Stacking-ensemble and ECA-EfficientNetV2 models, while the accuracies from six pre-trained models are inferior. In fact, the highest two performance for five metrics from Stacking-ensemble and ECA-EfficientNetV2 models are significantly different from six pre-trained models (p-value < 0.01). Although the best performance, the accuracy (99.21%), precision (99.23%), recall (99.25%), F1-score (99.20%) and AUC (99.51%), comes from the proposed ECA-EfficientNetV2 model, the differences for five metrics between Stacking-ensemble and ECA-EfficientNetV2 models are not significant. Comparing the standard deviations of five metrics from Stacking-ensemble and ECA-EfficientNetV2 models, those from the latter are relatively small, which stands the stability of the model.
      Table 11Performance comparison on Dataset X-ray
      ModelsDataset X-ray
      Accuracy (%)Precision (%)Recall (%)F1-score (%)AUC (%)
      EfficientNetV2-B085.23(±1.63)85.35(±1.61)85.26(±1.63)85.23(±1.65)90.77(±1.02)
      EfficientNetV2-B185.63(±2.00)85.68(±2.02)85.66(±2.00)85.61(±2.01)91.02(±1.25)
      EfficientNetV2-B283.31(±2.59)83.48(±2.63)83.34(±2.59)83.27(±2.58)89.58(±1.62) \
      EfficientNetV2-B385.78(±0.96)85.97(±0.99)85.84(±1.02)85.77(±0.96)91.11(±0.60)
      EfficientNetV2-S87.41(±0.47)87.50(±0.51)87.54(±0.49)87.42(±0.51)92.15(±0.31)
      EfficientNetV2-M88.65(±1.05)88.68(±1.01)88.73(±1.07)88.64(±1.05)92.88(±0.66)
      Stacking-ensemble98.71(±0.26)98.44(±1.05)98.75(±0.26)98.72(±0.26)99.20(±0.16)
      ECA-EfficientNetV299.21(±0.30)99.23(±0.30)99.25(±0.30)99.20(±0.31)99.51(±0.19)
      P-value0.00
      p<0.01
      0.00
      p<0.01
      0.00
      p<0.01
      0.00
      p<0.01
      0.00
      p<0.01
      low asterisklow asterisk p<0.01
      Fig. 12 shows examples of confusion matrix from EfficientNetV2-B0, EfficientNetV2-M, Stacking-ensemble, and ECA-EfficientNetV2 models. In EfficientNetV2-B0, 58 Bacterial Pneumonia are misclassified to Tuberculosis, and 42 Tuberculosis are misclassified to Bacterial Pneumonia. In EfficientNetV2-M, 40 Bacterial Pneumonia are misclassified to Tuberculosis, and 51 Tuberculosis are misclassified to Bacterial Pneumonia. Obviously, Bacterial Pneumonia and Tuberculosis are misclassified to each other from EfficientNetV2-B0 and EfficientNetV2-M models, while the wrong phenomenon is much improved in Stacking-ensemble and ECA-EfficientNetV2 models.

      Dataset CT

      The performances of 5 metrics among EfficientNetV2-B0, EfficientNetV2-B1, EfficientNetV2-B2, EfficientNetV2-B3, EfficientNetV2-S, EfficientNetV2-M, the proposed stacking-ensemble model, and the proposed ECA-EfficientNetV2 models for dataset CT were compared in Table 12. The accuracies from the above eight models are greater than 93%; the highest two are from Stacking-ensemble and ECA-EfficientNetV2 models. Actually, the same results could be found in precision, recall, F1-score and AUC. The five metrics from Stacking-ensemble and ECA-EfficientNetV2 models are statistically significant from the six EfficientNetV2 models (p-value < 0.01). The best performance, the accuracy (99.81%), precision (99.80%), recall (99.80%), F1-score (99.81%) and AUC (99.87%), comes from the proposed ECA-EfficientNetV2, which are not significantly different from the Stacking-ensemble model. Same as we found in Dataset X-ray, the standard deviations of five metrics from ECA-EfficientNetV2 model are relatively small, which stands the stability of the model.
      Table 12Performance comparison on Dataset CT
      ModelsDataset CT
      Accuracy (%)Precision (%)Recall (%)F1-score (%)AUC (%)
      EfficientNetV2-B094.54(±0.84)94.58(±0.83)94.54(±0.83)94.50(±0.85)96.36(±0.56)
      EfficientNetV2-B193.60(±1.67)93.66(±1.74)93.67(±1.61)93.55(±1.70)95.73(±1.11)
      EfficientNetV2-B294.41(±1.48)94.82(±2.04)94.47(±1.57)94.35(±1.49)96.28(±0.99)
      EfficientNetV2-B395.10(±0.73)95.15(±0.72)95.15(±0.68)95.12(±0.70)96.73(±0.49)
      EfficientNetV2-S96.34(±0.66)96.35(±0.67)96.35(±0.66)96.34(±0.67)97.56(±0.44)
      EfficientNetV2-M95.66(±0.65)95.67(±0.67)95.58(±0.57)95.65(±0.66)97.11(±0.43)
      Stacking-ensemble98.84(±0.51)98.87(±0.50)98.93(±0.54)98.92(±0.61)99.23(±0.34)
      ECA-EfficientNetV299.81(±0.27)99.80(±0.27)99.80(±0.28)99.81(±0.27)99.87(±0.18)
      P-value0.00
      p<0.01
      0.00
      p<0.01
      0.00
      p<0.01
      0.00
      p<0.01
      0.00
      p<0.01
      low asterisklow asterisk p<0.01
      Fig. 13 shows examples of confusion matrix from EfficientNetV2-B0, EfficientNetV2-M, Stacking-ensemble, and ECA-EfficientNetV2 models for Dataset CT. In EfficientNetV2-B0, 23 COVID-19 are misclassified to Normal, and 8 Normal are misclassified to COVID-19. In EfficientNetV2-M, 18 COVID-19 are misclassified to Normal, and 12 Normal are misclassified to COVID-19. In Stacking-ensemble, 7 COVID-19 are misclassified to Normal, and 1 Normal are misclassified to COVID-19. In ECA-EfficientNetV2, 2 COVID-19 are misclassified to Normal, and 0 Normal are misclassified to COVID-19. Obviously, COVID-19 and Normal are misclassified to each other from EfficientNetV2-B0 and EfficientNetV2-M models, while Stacking-ensemble and ECA-EfficientNetV2 models greatly improved the wrong phenomenon.

      Comparison with SOTA

      The results of the proposed Stacking-ensemble and ECA-EfficientNetV2 models are compared with the related studies using stacking ensemble models for COVID-19 diagnosis in Table 13. Most of the related studies worked on classification of 2 to 3 groups, while this study focuses on classification of 5 groups in X-ray and classification of 4 groups in CT. Most of the accuracies are greater than 90% by using stacking models, which represents the favorable performance of stacking method.
      Table 13Comparison of the proposed models with SOTA
      No.Study(s)DatasetArchitectureClassAccuracy
      1Ieracitano et al.

      (2022)
      • Ieracitano C.
      • et al.
      A fuzzy-enhanced deep learning approach for early detection of Covid-19 pneumonia from portable chest X-ray images.
      155 X-ray images.CovNNet281.00%
      2Khan et al.

      (2021)
      • Khan S.H.
      • et al.
      COVID-19 detection in chest X-ray images using deep boosted hybrid learning.
      6,448 X-ray images.DHL

      DBHL
      298.53%
      3Loey et al.

      (2022)
      • Loey M.
      • El-Sappagh S.
      • Mirjalili S.
      Bayesian-based optimized deep learning model to detect COVID-19 patients using chest X-ray image data.
      10,848 X-ray images.CNN+ Bayesian396.00%
      4Hu et al.

      (2021)
      • Hu T.
      • Khishe M.
      • Mohammadi M.
      • Parvizi G.R.
      • Taher Karim S.H.
      • Rashid T.A.
      Realtime COVID-19 diagnosis from X-Ray images using deep CNN and extreme learning machines stabilized by chimp optimization algorithm.
      10,848 X-ray images.CNN+ ELMs+ ChOA299.11%
      5Thaseen et al.

      (2022)
      • R S.
      • et al.
      An Efficient Hardware Architecture Based on An Ensemble of Deep Learning Models for COVID -19 Prediction.
      13,808 X-ray images.Ensemble CNN model399.00%
      6Kumar et al.

      (2021)
      • Kumar M.
      • Shakya D.
      • Kurup V.
      • Suksatan W.
      COVID-19 prediction through X-ray images using transfer learning-based hybrid deep learning approach.
      6,000 X-ray images.HCNN398.20%
      7Musallam et al.

      (2022)
      • Musallam A.S.
      • Sherif A.S.
      • Hussein M.K.
      Efficient Framework for Detecting COVID-19 and Pneumonia from Chest X-Ray Using Deep Convolutional Network.
      7,512 X-ray images.DeepChest396.56%
      8Mamalakis et al.

      (2021)
      • Mamalakis M.
      • et al.
      DenResCov-19: A deep transfer learning network for robust automatic classification of COVID-19, pneumonia, and tuberculosis from X-rays.
      6,000 X-ray images.DenResCov-19499.60%
      9Garg et al.

      (2022)
      • Garg A.
      • Salehi S.
      • la Rocca M.
      • Garner R.
      • Duncan D.
      Efficient and visualizable convolutional neural networks for COVID-19 classification using Chest CT.
      4,173 CT images.EfficientNet-B5298.45%
      10Lahsaini et al.

      (2021)
      • LAHSAINI I.
      • EL HABIB DAHO M.
      • CHIKH M.A.
      Deep transfer learning based classification model for covid-19 using chest CT-scans.
      4,986 CT images.DenseNet201+GradCam298.80%
      11Rahimzadeh et al.

      (2021)
      • Rahimzadeh M.
      • Attar A.
      • Sakhaei S.M.
      A fully automated deep learning-based network for detecting COVID-19 from a new and large lung CT scan dataset.
      63,849 CT images.ResNet50V2+ FPN298.49%
      12Qi et al.

      (2022)
      • Qi Q.
      • et al.
      Fully automatic pipeline of convolutional neural networks and capsule networks to distinguish COVID-19 from community-acquired pneumonia via CT images.
      10,000 CT images.UNet+ DenseNet121297.10%
      13Abdel-Basset et al.

      (2021)
      • Abdel-Basset M.
      • Hawash H.
      • Moustafa N.
      • Elkomy O.M.
      Two-Stage Deep Learning Framework for Discrimination between COVID-19 and Community-Acquired Pneumonia from Chest CT scans.
      9,593 CT images.GR-UNet+ CNN296.80%
      14Ye et al.

      (2022)
      • Ye Q.
      • et al.
      Robust weakly supervised learning for COVID-19 recognition using multi-center CT images.
      45,167 CT images.CIFD-Net291.19%
      15Balaha et al.

      (2021)
      • Balaha H.M.
      • El-Gendy E.M.
      • Saafan M.M.
      CovH2SD: A COVID-19 detection approach based on Harris Hawks Optimization and stacked deep learning.
      15,535 CT images.CNN+HHO+FCS+CSS+WSM299.33%
      16Ahamed et al.

      (2021)
      • Ahamed K.U.
      • et al.
      A deep learning approach using effective preprocessing techniques to detect COVID-19 from chest CT-scan and X-ray images.
      4,593 X-ray images.ResNet50V2(fine-tuning)496.45%
      3,000 CT images.399.01%
      17Kumari et al.

      (2021)
      • Kumari S.
      • Ranjith E.
      • Gujjar A.
      • Narasimman S.
      • Aadil Sha Zeelani H.S.
      Comparative analysis of deep learning models for COVID-19 detection.
      2,000 X-ray images.VGG16298.00%
      2,000 CT images.Xception283.00%
      18Ahsan et al.

      (2020)

      M. M. Ahsan, K. D. Gupta, M. M. Islam, S. Sen, Md. L. Rahman, and M. S. Hossain, “Study of Different Deep Learning Approach with Explainable AI for Screening Patients with COVID-19 Symptoms: Using CT Scan and Chest X-ray Image Dataset,” 2020, doi: 10.3390/make2040027.

      400 X-ray images.NasNetMobile2100.00%
      400 CT images.295.20%
      19Jia et al.

      (2021)
      • Jia G.
      • Lam H.K.
      • Xu Y.
      Classification of COVID-19 chest X-Ray and CT images using a type of dynamic CNN modification method.
      7,592 X-ray images.ResNet(fine-tuning)599.60%
      104,009 CT images.399.30%
      20Kassania et al.

      (2021)
      • Kassania S.H.
      • Kassanib P.H.
      • Wesolowskic M.J.
      • Schneidera K.A.
      • Detersa R.
      Automatic Detection of Coronavirus Disease (COVID-19) in X-ray and CT Images: A Machine Learning Based Approach.
      137 X-ray images.DenseNet121+ Bagging299.00%
      137 CT images.
      21Gour and Jain

      (2022)
      • Gour M.
      • Jain S.
      Automated COVID-19 detection from X-ray and CT images with stacked ensemble convolutional neural network.
      3,040 X-ray images.Ensemble CNN397.27%
      4,645 CT images.298.30%
      22Kamil et al.

      (2021)
      • Kamil M.Y.
      A deep learning framework to detect Covid-19 disease via chest X-ray and CT scan images.
      977 X-ray images.VGG19(fine-tuning)299.00%
      23 CT images.
      23Saygılı

      (2021)
      • Saygılı A.
      A new approach for computer-aided detection of coronavirus (COVID-19) from CT and X-ray images using machine learning methods.
      1,125 X-ray images.SVM385.96%
      3,228 CT images.K-ELM298.88%
      24Stacking-ensemble6,000 X-ray images.Stacking-ensemble598.71%
      4,800 CT images.498.84%
      25ECA-EfficientNetV26,000 X-ray images.ECA-EfficientNetV2599.21%
      4,800 CT images.499.81%
      Although the accuracy from our Stacking-ensemble model is slightly inferior than [
      • Hu T.
      • Khishe M.
      • Mohammadi M.
      • Parvizi G.R.
      • Taher Karim S.H.
      • Rashid T.A.
      Realtime COVID-19 diagnosis from X-Ray images using deep CNN and extreme learning machines stabilized by chimp optimization algorithm.
      ,
      • R S.
      • et al.
      An Efficient Hardware Architecture Based on An Ensemble of Deep Learning Models for COVID -19 Prediction.
      ,
      • Mamalakis M.
      • et al.
      DenResCov-19: A deep transfer learning network for robust automatic classification of COVID-19, pneumonia, and tuberculosis from X-rays.
      ,
      • Balaha H.M.
      • El-Gendy E.M.
      • Saafan M.M.
      CovH2SD: A COVID-19 detection approach based on Harris Hawks Optimization and stacked deep learning.
      ,
      • Ahamed K.U.
      • et al.
      A deep learning approach using effective preprocessing techniques to detect COVID-19 from chest CT-scan and X-ray images.
      ,

      M. M. Ahsan, K. D. Gupta, M. M. Islam, S. Sen, Md. L. Rahman, and M. S. Hossain, “Study of Different Deep Learning Approach with Explainable AI for Screening Patients with COVID-19 Symptoms: Using CT Scan and Chest X-ray Image Dataset,” 2020, doi: 10.3390/make2040027.

      ,
      • Jia G.
      • Lam H.K.
      • Xu Y.
      Classification of COVID-19 chest X-Ray and CT images using a type of dynamic CNN modification method.
      ,
      • Kassania S.H.
      • Kassanib P.H.
      • Wesolowskic M.J.
      • Schneidera K.A.
      • Detersa R.
      Automatic Detection of Coronavirus Disease (COVID-19) in X-ray and CT Images: A Machine Learning Based Approach.
      ,
      • Kamil M.Y.
      A deep learning framework to detect Covid-19 disease via chest X-ray and CT scan images.
      ,
      • Saygılı A.
      A new approach for computer-aided detection of coronavirus (COVID-19) from CT and X-ray images using machine learning methods.
      ]. As we have mentioned before, most of those related studies worked on classification of two groups, while this study is working on the classification of 4 groups in X-ray and 5 groups in CT. Even in this situation, the accuracies of our proposed Stacking-ensemble model are pretty close to the above studies. In addition, the performance of the proposed ECA-EfficientNetV2 model dominates most of the related studies. The accuracies for X-ray and CT from ECA-EfficientNetV2 model are close to 100%, which illustrates the great classification capability for multiple groups on chest diseases.
      To verify the performance of the two proposed Stacking-ensemble and ECA-EfficientNetV2 models, one more open public dataset COVID-CT (https://github.com/UCSD-AI4H/COVID-CT) was tested and the performance metrics were displayed in Table 14. There are 746 CT images including two groups of COVID-19 and Non-COVID-19 in dataset COVID-CT. Except the accuracy of 93.33% from Shaik and Cherukuri [
      • Shaik N.S.
      • Cherukuri T.K.
      Transfer learning based novel ensemble classifier for COVID-19 detection from chest CT-scans.
      ], the accuracies from other related studies are lower than 90%. The accuracies are 94.86% and 95.29% from our proposed Stacking-ensemble and ECA-EfficientNetV2 models, respectively, which are higher than all of the related studies compared in Table 14. Especially, the number of parameter in ECA-EfficientNetV2 model is 5,706,965, which is much less than those in [
      • He X.
      • et al.
      Sample-efficient deep learning for COVID-19 diagnosis based on CT scans.
      ,
      • Polsinelli M.
      • Cinque L.
      • Placidi G.
      A light CNN for detecting COVID-19 from CT scans of the chest.
      ,

      X. Yang, X. He, J. Zhao, Y. Zhang, S. Zhang, and P. Xie, “COVID-CT-Dataset: A CT Scan Dataset about COVID-19,” Mar. 2020, [Online]. Available: http://arxiv.org/abs/2003.13865

      ].
      Table 14Comparisons on COVID-CT dataset
      DatasetStudy(s)AccuracyPrecisionRecallF1-scoreParameter
      COVID-CTMishra et al.

      (2020)
      • Mishra A.K.
      • Das S.K.
      • Roy P.
      • Bandyopadhyay S.
      Identifying COVID19 from Chest CT Images: A Deep Convolutional Neural Networks Based Approach.
      88.30%86.70%
      Saqib et al.

      (2020)

      M. Saqib, S. Anwar, A. Anwar, L. Petersson, N. Sharma, and M. Blumenstein, “COVID19 detection from Radiographs: Is Deep Learning able to handle the crisis?,” no. June, pp. 1–14, 2020, [Online]. Available: www.preprints.org

      80.30%78.20%85.70%81.80%
      He et al.

      (2020)
      • He X.
      • et al.
      Sample-efficient deep learning for COVID-19 diagnosis based on CT scans.
      86.00%85.00%14,149,480
      Mobiny et al.

      (2020)

      A. Mobiny et al., “Radiologist-Level COVID-19 Detection Using CT Scans with Detail-Oriented Capsule Networks,” 2020, [Online]. Available: http://arxiv.org/abs/2004.07407

      84.00%
      Polsinelli et al.

      (2020)
      • Polsinelli M.
      • Cinque L.
      • Placidi G.
      A light CNN for detecting COVID-19 from CT scans of the chest.
      85.00%85.00%87.00%86.00%12,600,000
      Yang et al.

      (2020)

      X. Yang, X. He, J. Zhao, Y. Zhang, S. Zhang, and P. Xie, “COVID-CT-Dataset: A CT Scan Dataset about COVID-19,” Mar. 2020, [Online]. Available: http://arxiv.org/abs/2003.13865

      89.10%25,600,000
      Cruz

      (2021)
      • Hernández Santa Cruz J.F.
      An ensemble approach for multi-stage transfer learning models for COVID-19 detection from chest CT scans.
      86.00%89.00%85.00%
      Shaik and Cherukuri

      (2021)
      • Shaik N.S.
      • Cherukuri T.K.
      Transfer learning based novel ensemble classifier for COVID-19 detection from chest CT-scans.
      93.33%93.17%93.54%93.29%
      Stacking-ensemble94.86%94.79%94.83%94.84%
      ECA-EfficientNetV295.29%95.15%95.24%95.27%5,706,965

      Conclusion

      This study applies six pre-trained MobileNetV2 models on COVID-19 diagnosis, and stacks ensemble the above six models to achieve better classification results. The self-designed Stacking-ensemble and ECA-MobileNetV2 models were proposed to classify multiple chest diseases including COVID-19 in this study. Classification were executed on five groups using X-ray images and four groups using CT images. The experimental results of two proposed models were compared with six pre-trained MobileNetV2 models. The proposed Stacking-ensemble achieves accuracies of 98.71% and 98.84% on X-ray and CT datasets, respectively. With stability, the proposed ECA-MobileNetV2 achieves the highest accuracies of 99.21% and 99.81% on X-ray and CT datasets, respectively.
      The proposed Stacking-ensemble model is the first study ensembles the series of EfficientNetV2 models on multiple chest diseases including COVID-19 detection, which reduce prediction variance to the training data and improves overall classification performance when compared to any individual EfficientNetV2 model. As compared to the proposed ECA-MobileNetV2 model, the stacking model is computationally expensive; it takes around 1.5 hours in this experiment while the execution time is only 17 minutes in ECA-MobileNetV2 model. The major contribution of the study is the proposed ECA-MobileNetV2 model combines the advantages of EfficientNetV2 and ECA-Net to demonstrate superior classification performance with least variance and training time. The results of this study imply that the architecture of the proposed ECA-MobileNetV2 model can be used to assist radiologists in the X-ray and CT scans on multiple chest diseases including COVID-19.
      The 6,000 X-ray and 4,800 CT images used in this study were collected from five Kaggle datasets. More chest images are encouraged to achieve more robust classification performance. Since the related studies compared in the study were searched from Scopus and were limited, more comprehensive review is suggested in the future research.

      CRediT authorship contribution statement

      Mei-Ling Huang: Conceptualization, Formal analysis, Investigation, Methodology, Resources, Validation, Visualization, Writing – original draft, Writing – review & editing, Supervision. Yu-Chieh Liao: Methodology, Data curation, Software, Formal analysis, Investigation, Writing – original draft.

      Ethical approval

      This article does not contain any studies with human participants performed by any of the authors.

      References

      1. C. Pneumonia, “COVID-19,” vol. 2019, no. December, 2019.

        • Liu H.
        • Wang L.
        • Nan Y.
        • Jin F.
        • Wang Q.
        • Pu J.
        SDFN: Segmentation-based deep fusion network for thoracic disease classification in chest X-ray images.
        Computerized Medical Imaging and Graphics. 2019; 75: 66-73https://doi.org/10.1016/j.compmedimag.2019.05.005
        • Hou H.
        • et al.
        Artificial Intelligence Distinguishes COVID-19 from Community Acquired Pneumonia on Chest CT.
        Applied Intelligence. 2020; 2019 ([Online]. Available:): 1-5
        • Sharma N.
        • Sharma R.
        • Jindal N.
        Machine Learning and Deep Learning Applications-A Vision.
        in: Global Transitions Proceedings. 2021: 0-8https://doi.org/10.1016/j.gltp.2021.01.004
        • Ieracitano C.
        • et al.
        A fuzzy-enhanced deep learning approach for early detection of Covid-19 pneumonia from portable chest X-ray images.
        Neurocomputing. 2022; 481: 202-215https://doi.org/10.1016/j.neucom.2022.01.055
        • Khan S.H.
        • et al.
        COVID-19 detection in chest X-ray images using deep boosted hybrid learning.
        Computers in Biology and Medicine. 2021; 137104816https://doi.org/10.1016/j.compbiomed.2021.104816
        • Loey M.
        • El-Sappagh S.
        • Mirjalili S.
        Bayesian-based optimized deep learning model to detect COVID-19 patients using chest X-ray image data.
        Computers in Biology and Medicine. 2022; 142105213https://doi.org/10.1016/j.compbiomed.2022.105213
        • Hu T.
        • Khishe M.
        • Mohammadi M.
        • Parvizi G.R.
        • Taher Karim S.H.
        • Rashid T.A.
        Realtime COVID-19 diagnosis from X-Ray images using deep CNN and extreme learning machines stabilized by chimp optimization algorithm.
        Biomedical Signal Processing and Control. 2021; 68102764https://doi.org/10.1016/j.bspc.2021.102764
        • R S.
        • et al.
        An Efficient Hardware Architecture Based on An Ensemble of Deep Learning Models for COVID -19 Prediction.
        Sustainable Cities and Society. Feb. 2022; 103713https://doi.org/10.1016/j.scs.2022.103713
        • Kumar M.
        • Shakya D.
        • Kurup V.
        • Suksatan W.
        COVID-19 prediction through X-ray images using transfer learning-based hybrid deep learning approach.
        in: Materials Today: Proceedings. Dec. 2021https://doi.org/10.1016/j.matpr.2021.12.123
        • Musallam A.S.
        • Sherif A.S.
        • Hussein M.K.
        Efficient Framework for Detecting COVID-19 and Pneumonia from Chest X-Ray Using Deep Convolutional Network.
        Egyptian Informatics Journal. 2021; : 2022https://doi.org/10.1016/j.eij.2022.01.002
        • Mamalakis M.
        • et al.
        DenResCov-19: A deep transfer learning network for robust automatic classification of COVID-19, pneumonia, and tuberculosis from X-rays.
        Computerized Medical Imaging and Graphics. 2021; 94102008https://doi.org/10.1016/j.compmedimag.2021.102008
        • Garg A.
        • Salehi S.
        • la Rocca M.
        • Garner R.
        • Duncan D.
        Efficient and visualizable convolutional neural networks for COVID-19 classification using Chest CT.
        Expert Systems with Applications. 2022; 195116540https://doi.org/10.1016/j.eswa.2022.116540
        • LAHSAINI I.
        • EL HABIB DAHO M.
        • CHIKH M.A.
        Deep transfer learning based classification model for covid-19 using chest CT-scans.
        Pattern Recognition Letters. Dec. 2021; 152: 122-128https://doi.org/10.1016/j.patrec.2021.08.035
        • Rahimzadeh M.
        • Attar A.
        • Sakhaei S.M.
        A fully automated deep learning-based network for detecting COVID-19 from a new and large lung CT scan dataset.
        Biomedical Signal Processing and Control. Jul. 2021; 68https://doi.org/10.1016/j.bspc.2021.102588
        • Qi Q.
        • et al.
        Fully automatic pipeline of convolutional neural networks and capsule networks to distinguish COVID-19 from community-acquired pneumonia via CT images.
        Computers in Biology and Medicine. Feb. 2022; 141https://doi.org/10.1016/j.compbiomed.2021.105182
        • Abdel-Basset M.
        • Hawash H.
        • Moustafa N.
        • Elkomy O.M.
        Two-Stage Deep Learning Framework for Discrimination between COVID-19 and Community-Acquired Pneumonia from Chest CT scans.
        Pattern Recognition Letters. 2021; 152: 311-319https://doi.org/10.1016/j.patrec.2021.10.027
        • Ye Q.
        • et al.
        Robust weakly supervised learning for COVID-19 recognition using multi-center CT images.
        Applied Soft Computing. 2022; 116108291https://doi.org/10.1016/j.asoc.2021.108291
        • Balaha H.M.
        • El-Gendy E.M.
        • Saafan M.M.
        CovH2SD: A COVID-19 detection approach based on Harris Hawks Optimization and stacked deep learning.
        Expert Systems with Applications. Dec. 2021; 186https://doi.org/10.1016/j.eswa.2021.115805
        • Qi S.
        • et al.
        DR-MIL: deep represented multiple instance learning distinguishes COVID-19 from community-acquired pneumonia in CT images.
        Computer Methods and Programs in Biomedicine. Nov. 2021; 211https://doi.org/10.1016/j.cmpb.2021.106406
        • Ahamed K.U.
        • et al.
        A deep learning approach using effective preprocessing techniques to detect COVID-19 from chest CT-scan and X-ray images.
        Computers in Biology and Medicine. 2021; 105014https://doi.org/10.1016/j.compbiomed.2021.105014
        • Kumari S.
        • Ranjith E.
        • Gujjar A.
        • Narasimman S.
        • Aadil Sha Zeelani H.S.
        Comparative analysis of deep learning models for COVID-19 detection.
        Global Transitions Proceedings. 2021; 2: 559-565https://doi.org/10.1016/j.gltp.2021.08.030
      2. M. M. Ahsan, K. D. Gupta, M. M. Islam, S. Sen, Md. L. Rahman, and M. S. Hossain, “Study of Different Deep Learning Approach with Explainable AI for Screening Patients with COVID-19 Symptoms: Using CT Scan and Chest X-ray Image Dataset,” 2020, doi: 10.3390/make2040027.

        • Jia G.
        • Lam H.K.
        • Xu Y.
        Classification of COVID-19 chest X-Ray and CT images using a type of dynamic CNN modification method.
        Computers in Biology and Medicine. 2021; 134104425https://doi.org/10.1016/j.compbiomed.2021.104425
        • Kassania S.H.
        • Kassanib P.H.
        • Wesolowskic M.J.
        • Schneidera K.A.
        • Detersa R.
        Automatic Detection of Coronavirus Disease (COVID-19) in X-ray and CT Images: A Machine Learning Based Approach.
        Biocybernetics and Biomedical Engineering. 2021; 41: 867-879https://doi.org/10.1016/j.bbe.2021.05.013
        • Gour M.
        • Jain S.
        Automated COVID-19 detection from X-ray and CT images with stacked ensemble convolutional neural network.
        Biocybernetics and Biomedical Engineering. 2022; 42: 27-41https://doi.org/10.1016/j.bbe.2021.12.001
        • Kamil M.Y.
        A deep learning framework to detect Covid-19 disease via chest X-ray and CT scan images.
        International Journal of Electrical and Computer Engineering. 2021; 11: 844-850https://doi.org/10.11591/ijece.v11i1.pp844-850
        • Saygılı A.
        A new approach for computer-aided detection of coronavirus (COVID-19) from CT and X-ray images using machine learning methods.
        Applied Soft Computing. 2021; 105107323https://doi.org/10.1016/j.asoc.2021.107323
      3. M. Tan and Q. v. Le, “EfficientNetV2: Smaller Models and Faster Training,” Apr. 2021, [Online]. Available: http://arxiv.org/abs/2104.00298

      4. “COVID-19 Radiography Database | Kaggle.” https://www.kaggle.com/tawsifurrahman/covid19-radiography-database (accessed Nov. 04, 2021).

      5. “Chest X-Ray Images (Pneumonia) | Kaggle.” https://www.kaggle.com/paultimothymooney/chest-xray-pneumonia (accessed Nov. 04, 2021).

      6. “Tuberculosis (TB) Chest X-ray Database | Kaggle.” https://www.kaggle.com/datasets/tawsifurrahman/tuberculosis-tb-chest-xray-dataset (accessed May 06, 2022).

      7. “Large COVID-19 CT scan slice dataset | Kaggle.” https://www.kaggle.com/maedemaftouni/large-covid19-ct-slice-dataset (accessed Nov. 09, 2021).

      8. “COVID-19&Normal&Pneumonia_CT_Images | Kaggle.” https://www.kaggle.com/datasets/anaselmasry/covid19normalpneumonia-ct-images (accessed May 08, 2022).

      9. M. Tan and Q. V. Le, “EfficientNetV2: Smaller Models and Faster Training,” 2021.

      10. J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu, “Squeeze-and-Excitation Networks,” Sep. 2017, [Online]. Available: http://arxiv.org/abs/1709.01507

      11. Q. Wang, B. Wu, P. Zhu, P. Li, W. Zuo, and Q. Hu, “ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks,” Oct. 2019, [Online]. Available: http://arxiv.org/abs/1910.03151

        • Opitz D.
        • Maclin R.
        Popular ensemble methods: An empirical study.
        Journal of Artificial Intelligence Research. 1999; 11: 169-198https://doi.org/10.1613/jair.614
      12. M. A. Ganaie and M Hu. “Ensemble deep learning: A review,” arXiv preprint arXiv:2104.02395 (2021).

        • Sollich P.
        • Krogh A.
        Learning with ensembles: how over-fitting can be useful.
        in: Proceedings of the 8th International Conference on Neural Information Processing Systems. 1995: 190-196 (Pages)
        • Pourtaheri Z.K.
        • Zahiri S.H.
        Ensemble classifiers with improved overfitting.
        in: 1st Conference on Swarm Intelligence and Evolutionary Computation. 2016https://doi.org/10.1109/CSIEC.2016.7482130
        • Rajaraman S.
        • Kim I.
        • Antani S.K.
        Detection and visualization of abnormality in chest radiographs using modality-specific convolutional neural network ensembles.
        PeerJ. 2020; 8: e8693https://doi.org/10.7717/peerj.8693
        • Devnath L.
        • Fan Z.
        • Luo S.
        • Summons P.
        • Wang D.
        Detection and Visualisation of Pneumoconiosis Using an Ensemble of Multi-Dimensional Deep Features Learned from Chest X-rays.
        International Journal of Environmental Research and Public Health. 2022; 19: 11193https://doi.org/10.3390/ijerph191811193
        • Rajaraman S.
        • Cemir S.
        • Xue Z
        • Alderson P.
        • Thoma G.
        • Antani S.
        A Novel Stacked Model Ensemble for Improved TB Detection in Chest Radiographs.
        Chapter 1, Medical Imaging. CRC Press, Boca Raton2019
        • Devnath L.
        • Luo S.
        • Summons P.
        • Wang D.
        • Shaukat K.
        • Hameed I.A.
        • Alrayes F.S.
        Deep Ensemble Learning for the Automatic Detection of Pneumoconiosis in Coal Worker's Chest X-ray Radiography.
        Journal of Clinical Medicine. 2022; 11: 5342https://doi.org/10.3390/jcm11185342
        • Mishra A.K.
        • Das S.K.
        • Roy P.
        • Bandyopadhyay S.
        Identifying COVID19 from Chest CT Images: A Deep Convolutional Neural Networks Based Approach.
        Journal of Healthcare Engineering. 2020; 2020https://doi.org/10.1155/2020/8843664
      13. M. Saqib, S. Anwar, A. Anwar, L. Petersson, N. Sharma, and M. Blumenstein, “COVID19 detection from Radiographs: Is Deep Learning able to handle the crisis?,” no. June, pp. 1–14, 2020, [Online]. Available: www.preprints.org

        • He X.
        • et al.
        Sample-efficient deep learning for COVID-19 diagnosis based on CT scans.
        IEEE Transactions on Medical Imaging. 2020; XXhttps://doi.org/10.1101/2020.04.13.20063941
      14. A. Mobiny et al., “Radiologist-Level COVID-19 Detection Using CT Scans with Detail-Oriented Capsule Networks,” 2020, [Online]. Available: http://arxiv.org/abs/2004.07407

        • Polsinelli M.
        • Cinque L.
        • Placidi G.
        A light CNN for detecting COVID-19 from CT scans of the chest.
        Pattern Recognition Letters. 2020; 140: 95-100https://doi.org/10.1016/j.patrec.2020.10.001
      15. X. Yang, X. He, J. Zhao, Y. Zhang, S. Zhang, and P. Xie, “COVID-CT-Dataset: A CT Scan Dataset about COVID-19,” Mar. 2020, [Online]. Available: http://arxiv.org/abs/2003.13865

        • Hernández Santa Cruz J.F.
        An ensemble approach for multi-stage transfer learning models for COVID-19 detection from chest CT scans.
        Intelligence-Based Medicine. 2021; 5100027https://doi.org/10.1016/j.ibmed.2021.100027
        • Shaik N.S.
        • Cherukuri T.K.
        Transfer learning based novel ensemble classifier for COVID-19 detection from chest CT-scans.
        Computers in Biology and Medicine. 2021; 141105127https://doi.org/10.1016/j.compbiomed.2021.105127