Design of oral English teaching model based on multi-modal perception of the Internet of Things and improved conventional neural networks

External attention

Figure 2 illustrates the intricate architecture of the self-attention module, which serves as the fundamental feature processing method in ViT. It is important to note that we opted for the Transformer model instead of convolutional neural networks (CNN) or long short-term memory (LSTM) because we need to process distinct types of features, namely images and voices. CNN or LSTM alone would not suffice for this purpose. Furthermore, the Bert model is well-suited for handling large datasets. Firstly, we linear the fused features $F\in {\mathbb{R}}^{\tilde{N}\times d}$ ( $\tilde{N}$ represents the account of pixels, and $d$ refers to the dimension of the fused features) to $Query\in {\mathbb{R}}^{\tilde{N}\times d}$, $Key\in {\mathbb{R}}^{\tilde{N}\times d}$ and $Value\in {\mathbb{R}}^{\tilde{N}\times d}$. Then, we calculate the attention weighting matrix by $Query$ and $Key$. The specific calculation process is as follows:

(5) ${\alpha }_{i,j}=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(Query\otimes Ke{y}^{\mathrm{T}}\right)$where ${\alpha }_{i,j}$ Calculates a similarity score between $i$-th pixel and $j$-th pixel. In addition, we multiply. ${\alpha }_{i,j}$ By $Value$ and make a residual connection for $F$ to obtain the final output feature. The formula is as follows:

(6) $F_{out}=F+{\alpha }_{i,j}\otimes Value,F_{out}\in {\mathbb{R}}^{\tilde{N}\times d}$

In the aforementioned calculation process, the inherent correlation between samples is overlooked, leading to insufficient feature extraction for the model. Most pixels within an individual sample exhibit correlations with only a few other pixels. Enumerating these correlations results in redundant calculations and an excessive number of model parameters.

To address the limitations of self-attention, we introduce an external attention mechanism. This mechanism enhances the model by capturing correlations among intra-sample and inter-sample elements through two learnable external memory units. By incorporating external attention, we improve the model’s feature extraction capabilities while reducing the number of parameters required for computation.

The detailed structure is shown in Fig. 3. Firstly, we map the feature map $F\in {\mathbb{R}}^{\tilde{N}\times d}$ to the vectors $Q_E\in {\mathbb{R}}^{\tilde{N}\times d}$. Then, we use the product of a learnable external memory unit. $M_k\in {\mathbb{R}}^{S\times d}$ and $Q_E$ to obtain the attention weighting map $A_E$ By the regularization. The formula is as follows:

(7) $A_E=Norm\left(Q_E\otimes M_k\right)$

Subsequently, we apply the $A_E$ and another external memory component $M_v\in {\mathbb{R}}^{S\times d}$ To compute another more refined feature map. The final output $F_{out}$ It is obtained by performing residual operations on the input features. The formula is as follows:

(8) $F_{out}=F+A_E\otimes M_v$

Feature selection

In the sliding window method, it is possible for certain patches to contain only background information or capture a small portion of foreground objects. Decreasing the step size of the sliding window increases the occurrence of similar patches. Given the abundance of discriminative features with subtle variations in images and voices, including insufficient information from multiple patches can lead to redundant output features.

To tackle this issue, we propose the integration of a feature selection module. This module selectively removes discriminatory parts of images and voices to alleviate redundancy in the output. By incorporating the feature selection module, we can enhance the effectiveness of feature extraction by focusing on the most informative and discriminative aspects of the data. We cannot select the discriminative patches without attention to weight information for the original patch sequence. According to the attention weight of the previous $L-1$ layers, the feature selection module can select the discriminative features output by the $(L-1$)-th layer. Besides, we use them as the layer input to further refine the features. Finally, we regard the $(L-1$)-th layer’s output of the Transformer as shown in the formula:

(9) $Z_{L-1}=\left[Z_{L-1}^0,Z_{L-1}^1,\dots ,Z_{L-1}^N,Z_{L-1}^{N+1}\right]$

The attending weights of the previous $L-1$ layer are regarded as follows:

(10) ${\alpha }_l=\left[{\alpha }_l^0,{\alpha }_l^1,\dots ,{\alpha }_l^K\right],l=1,2,\dots ,L-1$

(11) ${\alpha }_l^i=\left[{\alpha }_l^{i_0},{\alpha }_l^{i_1},\dots ,{\alpha }_l^{i_N},{\alpha }_l^{i_{N+1}}\right],i\in \left[0,K\right]$where $K$ refers to the accounts of attention heads, and then, we feed the continuous multiplication operation into the feature selection module to integrate the attended weightings for the previous $L-1$ layers, which can be shown in the formula:

(12) $\alpha =\prod _{l=0}^{L-1}{\alpha }_i$where $\alpha$ records the transfer process of attention weight from the first layer to the $(L-1$)-th layer, we choose the $k$ indexes. $\left[A_1,A_2,\dots ,A_K\right]$. They correspond to the maximum value of each attention head. Finally, the features are chosen to input them to the $L$-th layer according to the index. The selected sequence can be expressed as follows:

(13) $Z=\left[Z_{L-1}^0,Z_{L-1}^{A_1},Z_{L-1}^{A_2},\dots ,Z_{L-1}^N,Z_{L-1}^{N+1}\right]$where $Z$, regarded as the input of the final layer, drops too many invalid features extracted to avoid the redundancy of the last output features.

Fused multivariate loss function

To address the subtle differences between different classes of images and voices, as well as the significant differences within the same class, we propose a fused multivariate loss function that optimizes from multiple perspectives. This approach aims to overcome the weak bias induction ability of Transformer models.

In our proposed approach, we utilize a combination of cross-entropy loss and contrastive loss. The cross-entropy loss is employed to capture the noticeable inter-class differences. On the other hand, the contrastive loss serves to increase the dissimilarity between features of subclasses while reducing the disparity among features of the same subclass. In this article, we maintain the calculation of the cross-entropy loss based on the class patch output from the last Transformer. Simultaneously, we calculate the contrastive loss using the same patch. The calculation process can be expressed as follows:

(14) ${\tau }_1={\displaystyle \frac{1}{N_B^2}\sum _i^{N_B}\left[\sum _{j:y_i=y_i}^{N_B}\left(1-\mathrm{c}\mathrm{o}\mathrm{s}\left(Z_i,Z_j\right)\right)+\sum _{j:y_i\ne y_i}^{N_B}max\left(\left(\cos \left(Z_i,Z_j\right)-\alpha \right),0\right)\right]}$where $N_B$ Is the size of the batch, $Z_i$ It is the class patch output of the $i$-th image through the model, which is also the final feature representation. Besides, $\mathrm{c}\mathrm{o}\mathrm{s}\left(Z_i,Z_j\right)$ represents the cosine similarity of $Z_i$ and $Z_j$, which will play a role in the contrast loss when it is greater than the hyperparameter $\alpha$. Through backpropagation, ${\tau }_1$ can extend the feature representation between different subcategories and reduce the feature representation within the same subcategory, alleviating the classification difficulty caused by small between-class and large within-class differences.

In addition to the previously mentioned loss functions, we recognize the importance of inductive bias in affecting feature extraction within the Transformer model. Given that convolutional neural networks (CNNs) possess a strong inductive bias, we introduce a distillation loss from the CNN to enhance the feature extraction process. In this article, we incorporate the distillation loss as part of the total loss function. We introduce a distillation patch, which is added after the input patch sequence. Similar to the class patch, the distillation patch interacts with other patches in multiple Transformer layers, and the features of the image are subsequently aggregated. However, unlike the class patch, the distillation patch reproduces the predicted labels generated by the Teacher model (CNN), rather than the actual labels.

To achieve the distillation loss, we calculate the Kullback–Leibler (KL) divergence based on the combination of the labels calculated by the distillation patch and the output labels of the Teacher Model (CNN). By incorporating the distillation loss, we enable the Transformer model to learn the inductive bias from the CNN, thereby significantly improving the feature extraction process. As the partial module of our loss function, the Student model is guided to perform backpropagation by the loss above. The specific calculation method is as follows:

(15) ${\tau }_2={\phi }^{2}KL\left(\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{Z_{stu}}{\phi }}\right),\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{Z_{tea}}{\phi }}\right)\right)$where $Z_{stu}$ Is the output of the Logist function when using a distillation patch for classification, $Z_{tea}$, is the output of the logits function of the Teacher model, and $\phi$ represents the distillation temperature.

To summarize, we incorporate three loss functions—cross-entropy loss, contrast loss, and distillation loss—from different perspectives. These losses are fused together to enhance the model’s ability to distinguish between subclasses, reduce differences within the same subclass, and provide the model with inductive bias. This fusion process results in more refined and discriminative output features.

The cross-entropy loss and contrast loss are calculated based on the same class patch, and thus, they are considered as a part of the total loss in the fusion process. On the other hand, the distillation loss is treated as a separate component. The fusion method we employ can be described as follows:

(16) $\tau =\left(1-\gamma \right)\left({\tau }_{CE}\left(y^{\mathrm{\prime }},y\right)+{\tau }_1\left(Z\right)\right)+\gamma {\tau }_2$where ${\tau }_{CE}\left(y^{\mathrm{\prime }},y\right)$ Is the cross-entropy loss for the predicted $y^{\mathrm{\prime }}$ As well as the true $y,$ according to the class patch, and $\gamma$ is the hyperparameter.

Datasets

The dataset of oral English teaching scoring includes 50,000 images and corresponding voices, which are from the videos of cram school on the IoT. We choose 40,000 pairs of image and voice as the training data, 5,000 pairs as validation, and 5,000 as test sets. In addition, we apply horizontal flip, vertical flip, and auto-augment to expand the dataset, which can avoid overfitting the model due to too little data.

Implement details

We implemented our experiments with the i7-12900k Cpu, 3080ti Gpu and the Pytorch deep learning framework. The training settings of the scoring model for oral English teaching based on the multi-modal perception of the Internet of Things are represented in Table 1.

We choose accuracy as the evaluation metric, which can be presented as:

(17) $$Accuracy={\displaystyle \frac{I_c}{I_{total}}}$$where $I_c$ refers to the accounts of correct classification images and $I_{total}$, refers to the counts for the total images.

In this article, we utilize a 12-layer structure for the Transformer model. The number of patches has a significant impact on the model’s parameter count, inversely proportional to the patch size and proportional to the input image’s resolution. To ensure sufficient data for the model evaluation and training convergence, we use a resolution of 448 × 448 for input images. During the training phase, we employ random cropping, while central cropping is used during the testing phase. The patch size of 16 × 16 from the original ViT model is retained, and the sliding window step is set to 12. In the training phase, we set the hyperparameter α to 0.4. For optimization, we employ stochastic gradient descent (SGD) with a momentum setting of 0.9. The batch size is set to 32, and the initial learning rate for training is 0.03. We also incorporate cosine annealing to control the learning rate’s decline.

To gather information, we collect images, audio, and video from various sensors in the IoT. We then perform feature extraction using ResNet, ensuring that the feature extraction model satisfies the aforementioned training parameters. Subsequently, we concatenate the extracted image features and voice features before inputting them into the Transformer model. Finally, these features are fed into the Transformer to obtain the desired results.

Experiments results

Ranks of classification division

Before performing the experimental performance comparison, we must artificially set some scoring ranks and unify the classification criteria. We also assign accurate classes to the features in the dataset. To realize it, we artificially divide the scoring positions into excellent, good, medium, pass, and poor in Table 2.

Table 2:

Ranks of scoring.

Ranks	Values
Excellent	90~100
Good	75~89
Medium	60~75
Pass	40~59
Poor	<40

DOI: 10.7717/peerj-cs.1503/table-2

Furthermore, to establish the ground truth and evaluate the performance of our model, we manually scored and classified 50,000 data groups. This process allowed us to obtain the score distribution and statistics for the entire dataset, as depicted in Figs. 4 and 5.

From the analysis of Figs. 4 and 5, we observed that the majority of the data in the dataset fell into the categories of “Excellent” and “Medium,” while the classes of “Ideal,” “Pass,” and “Poor” accounted for a smaller portion. Based on this observation, we determined that using five ranks for classification would be suitable. The distribution of ranks shown in Fig. 4 was deemed optimal. It is important to note that the number of samples in the “Excellent” and “Poor” ranks is relatively low, making further subdivisions impractical.