The neural machine translation models for the low-resource Kazakh–English language pair

Synthetic corpora formation approaches and transfer learning

When dealing with low-resource languages, it is important to use approaches that increase the corpora size to reach a higher quality of MT. There are generally three approaches to enhancing the size of the corpora and the performance of the models: forward translation (FT) (Zhang & Zong, 2020), backward translation (BT) (Abdulmumin, Galadanci & Isa, 2020), and transfer learning (TL) (Wu et al., 2022).

BT is generally a quality control method for translations. In this method, the monolingual data of the target language is translated back to its source language by an independent translator or a translation system. Then the generated synthetic data is added to the existing parallel corpora to increase their size. BT is also used to check the quality of the original parallel corpora. This way, the translated texts are checked by comparing them with the original variant. It can help identify errors, ambiguity, or confusion that may arise with the translation. However, the synthetic data received by BT is usually much noisier than the original data, and due to the nature of the language, a 100% perfect match is generally unattainable by this method. Since the goal of translation is to preserve the meaning of the translated text, two translators can offer two different translations that retain the original text’s meaning. BT is a three-stage translation quality control method that includes the translation of the completed translation back into the original language, the comparison of this new translation with the original text, and the reconciliation of any material differences between them.

Generally, the BT approach is implemented by the following series of steps:

• – There is a translation platform– $T_p$ and the target data– $Y=\{y{\}}_{i=1}^N$, where $y$ is a target text and $N$ is the total number of texts;

• – The $T_p$ is used on the target data $Y$ to get the source data $X=\{x{\}}_{i=1}^N$, where $x$ is a source text and $N$ is the total number of texts;

• – The newly generated parallel corpora $D^{\ast }$ is received by combining $X$ and $Y$;

• – Then the initially existing parallel corpora $D$ are added to $D^{\ast }$.

The scheme of the BT approach is shown in Fig. 1.

FT is the opposite approach to BT, where the monolingual data of the source language is translated to the target language by the translation system. While this method allows synthesizing parallel corpora, it is also used for quality control of the existing parallel corpora in the same way BT did.

The FT approach is implemented by the following series of steps:

• – There is a translation platform– $T_p$ and the source data– $X=\{x{\}}_{i=1}^N$ where $x$ is a source text and $N$ is the total number of texts.

• – The $T_p$ is used on the source data $X$ to get the target data $Y=\{y{\}}_{i=1}^N$, where $y$ is a target text and $N$ is the total number of texts.

• – The newly generated parallel corpora $D^{\ast }$ are received by combining $X$ and $Y$.

• – Then the initially existing parallel corpora $D$ are added to $D^{\ast }$.

The scheme of the FT approach is shown in Fig. 2.

Another technique commonly used in the low-resource MT scenario is TL. This approach uses a high-resource language pair to train the model, which is then applied to the low-resource pair. The proposed method enhances the performance of a low-resource model trained from scratch. Here we have $L_1\to L_3$ as a high-resource language pair and $L_2\to L_3$ as a low-resource pair. $L_1$ and $L_2$ are source languages, and $L_3$ is a target language for both pairs. $M_{L_1\to L_3}$ is a parent model trained on a $L_1\to L_3$ pair. The parameters of this model are used to fine-tune a $M_{L_2\to L_3}$ model that uses a $L_2\to L_3$ pair. The scheme of the TL technique is shown in Fig. 3.

The presented methods are directed at solving the problem of low-resource languages. In addition to dealing with the low-resource problem, the NMT architectures are also very important in increasing the performance of the trained models.

Corpora formation

The initial KZ–EN parallel corpora were parsed from Kazakhstan’s official bilingual government Internet sources (The Republic of Kazakhstan, 2023; Strategy, 2022; Primeminister, 2022) in the amount of 205,000 sentences with the use of the Bitextor tool. The crawled sentences were preprocessed (normalized, cleaned from extra symbols, characters, and empty strings, and aligned). Then the monolingual texts in the Kazakh language were gained from the scientific articles published in the Kazakh language in scientific journals (Kalekeyeva & Litvinov, 2021; Sapakova, 2022) that were taken because they had high-quality texts edited by the authors and checked by the editors. The texts from the scientific articles were also preprocessed (irrelevant symbols, characters, and words were deleted, and empty sentences were removed). The available open-source parallel corpora from the OPUS website were not considered for training because they included very restricted thematic unrelated to political or scientific topics, and the quality of texts was generally poor. The monolingual Kazakh texts in the amount of 205,000 sentences from scientific journals were translated with the use of the Promt MT system (the FT approach was applied). This platform supports different pairs of languages and is good for the KZ–EN translation.

After the crawled and synthetically generated KZ–EN corpora are gained, they are combined together to form the full corpora of 380,000 parallel sentences. The corpora are available on GitHub (Karyukin, 2022). The scheme of the parallel corpora formation is shown in Fig. 4.

The combined corpora went again through the quality-checking phase, where the sentences containing only mathematical symbols and other characters and sentences with poor translation were removed.

The NMT architectures

The fully formed KZ–EN parallel corpora are used for training NMT models. The NMT architectures have been intensively developing for the last decade. The traditionally used NMT architecture was Seq2Seq (Sindhu, Guha & Singh Panwar, 2022) for translating from one language to another. It is a deep learning model that takes a sequence of elements (words, letters, time series, etc.) and outputs another sequence of elements. The model consists of two main components: an encoder and a decoder. First, the encoder processes all inputs by transforming them into a single vector called a context. The context contains all the information that the encoder discovers from the input. Then, the vector is sent to the decoder, which creates an output sequence. Because the task is sequence-based, both the encoder and decoder tend to use some form of RNN, such as LSTM, GRU, etc. The hidden state vector can be of any size, although in most cases, it is taken as a power of two (layer) and length (128, 256, 512, 1,024), which can somehow reflect the complexity of the entire sequence. The scheme of the seq2seq model is shown in Fig. 5.

The RNN encoder and decoder model (Sharma et al., 2021) is designed to process a sequence of input texts and give a sequence of output texts. The hidden states contain loops where output at a one-time step becomes an input at another time step, defining a memory form. Therefore, the whole architecture of an RNN includes the following elements:

• – Inputs are words of a sentence in one language.

• – Embedding layers convert all words to corresponding vectors.

• – The encoder applies the context of word vectors from a previous time step to the current word vector.

• – The decoder transforms the encoded input into the translation sequence.

• – Output is a translated sentence.

The RNN architecture is shown in Fig. 6 (Sharma et al., 2021). The RNN architecture is also modified to model the history and future generated context (Shanmugavadivel et al., 2022). The neurons are split into the parts responsible for the forward and backward directions. At the same time, outputs from the forward states are not connected to inputs of the backward states, and it is also applied to the reverse as well. This designed architecture transforms into a unidirectional RNN without the backward states. The BRNN architecture is shown in Fig. 7 (Shanmugavadivel et al., 2022).

Recently, a new MT architecture called the Transformer model (Wan et al., 2022) was introduced. In this implementation, the RNN layers are replaced with the attention ones. As a result, the Transformer does not process input text sentences in sequential order. Instead, the self-attention mechanism identifies the context that gives meaning to an input sequence. Therefore, it provides more parallelization and reduces the training time.

The Transformer architecture is shown in Fig. 8 (Wan et al., 2022).

Tokenization techniques

The texts of the parallel corpora go through tokenization before being trained by NMT models. In NLP, tokenization is dividing the raw text into smaller units called tokens. The widely used tokenization techniques are word, character, and Byte-Pair Encoding (BPE).

In word tokenization, sentences are split by space into words, and these words are considered to be tokens. For example, the sentence “This model will be trained” is tokenized by words as {‘This’, ‘model’, ‘will’, ‘be’, ‘trained’}. Although this technique is simple and clear, it requires to have a huge vocabulary to train a good MT model, or the words that are not present in it will be translated as <UNK>.

The character tokenization technique is designed to solve the problem of a huge vocabulary for word tokenization. In this technique, the word is split by each character. For example, “Model” is tokenized as {‘M’, ‘o’, ‘d’, ‘e’, ‘l’}. The disadvantage of this tokenization technique is the requirement of large computational resources to train an MT model.

Subword tokenization is a technique between word and character tokenization methods. It splits words into smaller subwords. For example, “Models” is split into “Model” and “s”. “Training” is split into “Train” and “ing”. One of the most popular subword tokenization algorithms is Byte-Pair Encoding (BPE), where the data compression algorithm is utilized.

Model evaluation

The quality of MT models is evaluated with the BLEU (Mouratidis, Kermanidis & Sosoni, 2020), WER (Stanojević et al., 2015), and TER (Bojar, Graham & Kamran, 2017) metrics, which show a high correlation with human quality ratings. These metrics’ values range from 0 to 1 (from 0% to 100% if the percentage measure is used). The BLEU metric is based on the idea that “the closer the MT to the professional human translation, the better it is.” It is computed as the ratio of the number of words of the translation candidate found in the reference translation to the total number of words of the candidate:

(1) $$BLEU=Brevity\mathrm{\_}Penalty\times \prod _{n=1}^{N}p{}_n^{w_n},$$where $p_n$ are the precision values of the N-gram; N is the maximum level of granularity (unigram, bigram, trigram, etc.); $w_n$ are the weights of the N-gram; Brevity_Penalty is the measure that penalizes sentences that are too short:

(2) $$Brevity\mathrm{\_}Penalty=\{\begin{array}{c}1,c>r\hfill \\ e^{(1-r/c)},c\le r\hfill \end{array},$$where $c$ is the length of the candidate translation and $r$ is the length of the reference.

The closer the BLEU score value to 1 (100%), the higher the translation quality is.

The WER metric finds the distance between a translation candidate and its reference. The minimum number of edit operations (insertion, substitution, and deletion) is divided by the number of words in the reference:

(3) $$WER(c,r)={\displaystyle \frac{min(e)}{r},}$$where $r$ is the reference; $c$ is the candidate; $e$ is the number of edit operations. The quality of the WER metric is evaluated opposite to the BLEU metric. The closer the WER score value to 0(0%), the better the translation is.

The WER metric has a significant drawback because it does not consider shifts of words. The TER metric improves this case by considering shifts in sentences. Thus, it is calculated as follows:

(4) $$TER(c,r)={\displaystyle \frac{min(e+shifts)}{r},}$$where $r$ is the reference; $c$ is the candidate; $e$ is the number of edit operations; $shifts$ is the number of shift operations. The quality of the TER metric is evaluated similarly to the WER metric. The closer the TER score value to 0-(0%), the better the translation is.

Training MT models

The formed full parallel KZ–EN corpora were moved to the training phase with the described NMT models. The OpenNMT framework, an open-source NMT application, was utilized for this purpose. This tool contains many different architectures whose parameters are easy to configure. The RNN, BRNN, and Transformer models were tuned to train MT models. The sentences were tokenized with word and BPE subword tokenization techniques. The character tokenization technique was not implemented because it used large computational resources. The RNN, BRNN, and Transformer MT models had the following parameters for the training phase (Table 2). The full corpora were divided into 90% for training, 5% for validating, and 5% for testing phases, leaving the largest part for training to maximize the performance of the models. The configuration of the OpenNMT framework is available on GitHub (Karyukin, 2022).

The experiments were conducted on the workstation with the following specifications: Core i7 4790K, 32 GB RAM, 1 TB SSD, and NVIDIA Geforce RTX 2070 Super. In addition, for training the Transformer models, the second GPU, NVIDIA Geforce GTX 1080, was added to the bundle with RTX 2070 Super to launch those heavy models. The values of each of these metrics for every architecture with word tokenization and BPE are shown in Table 3.

The models showed a good performance for the KZ–EN corpora, comparable with the high-resource trained models. The examples of the translation results of test sentences are shown in Table 4. Here, Source is the sentence in the Kazakh language; Target is the sentence in the English language; Predict is the translation of test sentences by the trained NMT models (RNN, BRNN, and Transformer).

Comparing the values of BLEU, WER, and TER metrics from Table 2 and translations from Table 3, it is seen that the RNN and BRNN models give a more precise translation than the Transformer model. The BPE tokenization technique showed better metrics scores and translation than the word tokenization technique. Nevertheless, analyzing the results of other works devoted to the NMT for the KZ–EN corpora, the improvements in the metrics scores are noticeable. Having the corpora mostly equal in size to the corpora of 400,000 parallel KZ–EN sentences in Tukeyev, Karibayeva & Abduali (2018) made it possible to achieve the best BLEU score of 0.16. The article (Toral et al., 2019) had only 67,000 parallel KZ–EN sentences, and the best BLEU score was 0.28. Despite the difference in the size and quality of corpora used in those works, the experimental results in this research show the importance of the corpora of parallel sentences and NMT architectures for achieving the best KZ–EN MT results. A very important note is that the RNN and BRNN models’ scores were higher than the metrics’ values of the Transformer model for the gathered corpora. The BPE tokenization technique outperformed the word tokenization for all the used models. All these NMT models can be used in the MT application currently under development by our research group. Now, the BRNN model with the BPE tokenization technique is adopted there (Fig. 9).

Although the evaluation metrics and translation samples in Tables 2 and 3 provide opportunities for comparing the results among sentences and different NMT models, it would also be preferable to have other Kazakh–English corpora with a number of sentences of around 500,000 to 1 million to get closer to the high-resource scenarios to have an even broader analysis. It is the task of future works.