2.1 NMT

The corpus-based (also known as data-driven) approach of NMT introduces RNN-based encoder-decoder architectures, where seq-2-seq learning is achievable by tackling variable length phrases of source-target sentences [³, ²⁶].

To learn the long-term features of the source and target words for encoding and decoding, long short-term memory (LSTM) has demonstrated remarkable performance in this case. When encountering too lengthy sentences, it is unable to encode all the necessary information.

For that reason, the attention mechanism has been introduced in NMT [³, ²⁶] that enables the decoder to take into account various segments of the source sequence during various decoding steps.

In the encoder-decoder based NMT, the encoder is responsible for the encoding of input sequences sr1,sr2…srn and generates a vector U.

Whereas, the decoder decodes the output tr1,tr2…trm using computation of condition probability, as given in Eq. (1):

Using Eq. (2), the value of ato correlates to the frequency of time steps in the input sentence. Therein, in the source side (hi) and target side (ho), the series of hidden states are computed, which are finally correlated to produce the attention vector ato:

The general estimate of score function defined in Eq. (3) is considered in this work for the preliminary experiments of the baseline system:

Then, the context vector cl is computed by using the hidden states average input weights with the attention vector. The attentional hidden vector is computed using Eq. (4) by the concatenation of ho and ct:

Finally, the softmax layer is included to the vector h′0 using Eq. (5) to obtain the predicted target sequence:

The disadvantages of RNN-based NMT in terms of parallelization and long-term dependencies are tackled by introducing transformer-based NMT [⁴²]. The primary idea behind the transformer model is to make use of the self-attention mechanism, an attention mechanism found inside the encoder.

Each token position is encoded by the transformer model, and self-attention is employed to connect two different tokens that aid in parallelization to quicken learning. The self-attention, also known as multi-head attention, computes attention several times.

The encoder-decoder architecture of transformer-based NMT contains six identical attention layers that are placed on stack of each other. The position of the input sequence is encoded and embedded to combine the sequence of tokens prior to feeding the sequence into the network.

The encoder consists of a point-wise connected feed-forward network layer and multiple headed attention layer. Whereas, the decoder comprises three layers and two of these layers are identical to the encoder.

The another multi-head attention layer is the third layer of the decoder that focuses to attend the output sequence a headed by the encoder. Here, the attention is calculated by considering the dot product of the input and utilizing a softmax function to get the weight of each token at a given position using Eq. (6):

To compute the attention, input vectors such as query (Q), key (K) with dimension dk, and value (V) are used. The advantage of using multi-head (MHD) attention in the transformer model over single-head attention is that it allows you to deal with different word representations through multiple positions. As shown in Eq. (7) and (8), the number of parallel attention heads accounts for h=8:

where the parameter matrices WiQ∈Rdmodel×dk, WiK∈Rdmodel×dk, and WiV∈Rdmodel×dr.

2.2 Related Work on English–Assamese MT

In literature review of the MT for English-Assamese pair, it is noted that the researchers are working on the dataset preparation to overcome the dataset’s scarcity for such a low-resource pair [⁴, ¹⁴, ²³, ³⁷]. The authors of [⁴] build a phrase-based SMT translation system via preparation of a small En-As parallel corpus of 14,371 sentences.

In our previous work [²³], a parallel corpus, namely, EnAsCorp1.0 [²³] is developed, and it contains 210,315 parallel sentences. And, the same has been used to build baseline models for En-As pair translation using the phrase-based SMT and RNN-based NMT.

Then in the previous work [²⁴], we have explored different NMT models (RNN and transformer) with data augmentation approach and attains better results on the same test set [²³] for En-As pair translation.

Moreover, a parallel corpora, namely, Samanantar [³⁷] that contains 11 Indian languages with English, and it includes 141,353 English-Assamese parallel sentences. Also, they [³⁷] implemented transformer-based NMT model for the En-to-Indic and Indic-to-En.

It is noted that all the prior works that have been conducted on this English-Assamese MT are not domain specific. In this work, we have prepared domain specific English-Assamese parallel corpus and utilized parallel corpus of EnAsCorp1.0 and Samanantar to enhance the translational performance for both forward and backward directions of translation. We have addressed data scarcity and word order divergence issues via data augmentation and guided alignment concept.

5.1 Data Augmentation

We have tackled data scarcity problem via data augmentation in two-ways: augmenting phrase-pairs and utilizing synthetic parallel data without modifying the NMT model architecture.

Following the strategy [³⁸], phrase-based SMT is trained on original parallel data using Moses^fn toolkit and extracted phrase pairs from the generated phrase table.

However, in our previous work [²⁴], it is noticed that the extracted phrase pairs contain wrong alignment phrases [²⁰].

Therefore, we have extracted phrase pairs by considering different translation probabilities (Setp≥0.5/Setp=1/Setall) of target phrases given source phrases following [³⁸, ²⁴] and observed that the translation accuracy with augmentation of extracted phrase pairs having translation probability Setp≥0.5 are higher.

Therefore, we have considered phrase pairs with translation probability Setp≥0.5 and the data statistics are reported in Table 6.

Further, to expand the parallel corpus, monolingual data is used to generate synthetic parallel data following BT strategy [³⁹, ²⁴]. However, it is observed that the translational accuracy with augmented data is lower than the without augmented one.

Therefore, following our previous work [²⁴] a two-step solution is used [¹]. First, pretrain the NMT model with synthetic data and “original parallel corpus + phrase pairs” and then fine-tune or reload it on the “original parallel corpus + phrase pairs”.

As a result of this , the final model initializes the parameters from the pretrained model that gains the training performance when the “original parallel corpus + phrase pairs” is utilized. We have used As-to-En transformer-based NMT model to generate synthetic parallel data using Assamese monolingual sentences since it gives higher translation accuracy, as shown in Table 4, 5.

To examine the effect of augmented synthetic parallel data, we have performed a series of experiments like our previous work [²⁴] on the ratio of parallel and synthetic corpora. It is noticed that 1:3 + phrase pairs attain higher translation accuracy for As-to-En and similar observation is found in case of En-to-As with 1:4 + phrase pairs and therefore, we have reported these results in Section 6.

5.2 Prior Alignment and Pretrained LM

The word order or token position of English is different from Assamese [²⁴] that leads to word-order divergence issue. In this work, we have attempted to extract token alignment information from the En-As bi-text data and feeded into NMT to enhance En-to-As and As-to-En directions of translation. In [³⁰], FastAlign tool is used to extract the token alignment information from the parallel data and adopted the guided alignment concept in the transformer-based NMT [¹⁰].

In [³⁰, ¹⁰], the optimization criteria for training the baseline transformer model [⁴²] is presented in Eq. 10, where T denotes the number of target tokens, p represents the output probability distribution, and ri,j indicates j−th the token in the dictionary is the true value at the i−th position in the target sentence.

The modified optimization criteria is represented in Eq. 11, where a pair of source-target sentences of length K and T, respectively, and a prior alignment set:

It takes randomly the output of just a head from the fifth decoder layer and project it into T target token probability distribution over K corresponding source token.

It compares the probability distributions qij with the reference probability generated from prior alignments through cross-entropy. The symbol ai,j represents the i−th target token is properly aligned with the j−th source token.

Both L1 and L2 are combined in Eq. 12 [¹⁰, ³⁰] which is the sum of cross-entropy for tokens and alignment weights of source-target sentences:

where, λ is a weighted cross-entropy for alignments (a hyperparameter), the authors [³⁰] considered 0.05. For comparative analysis, we also considered FastAlign to extract alignment information and, however, we have considered weighted alignment λ=0.03 since it yields the lowest training cost.

In this work, we have proposed to use SimAlign^fn [¹²] tool to extract the token alignment information. The SimAlign is a word alignment tool that uses static and pretrained multilingual language model (mBERT) based contextualized embeddings.

It uses sub-word (BPE) level processing in three various methods: Argmax, Itermax and Match to obtain the alignment information. The basic difference among these three methods is Argmax finds a local optimum and Itermax uses greedy algorithm, whereas Match finds a global optimum via maximum-weight maximal matching technique [¹²].

Although, we have extracted alignment information using these three methods, reported only Match-based SimAlign, since it shows higher translational performance in the NMT. We have used the two-step process to construct the alignments. First, extract the alignment information from both the forward and backward direction, i.e., En-to-As and As-to-En.

Then, combine the bidirectional alignments using the grow-diagonal heuristics of [¹⁷]. For the comparative analysis, we also considered extracted unidirectional (En-to-As or As-to-En) alignment information (as reported in Section 6.2).

It is noticed that the backward direction, i.e., As-to-En translation attains a higher score than that of En-to-As translation. Therefore, we have proposed to use the backward/reverse direction (As-to-En) of alignment information in the forward direction (En-to-As) translation using a simple two-step solution.

First, we reverse the extracted alignment information of the backward direction and then sort them to obtain the alignment information of forward direction.

The Marian [¹³] toolkit is employed to uses the source-target prior alignment information in the training process of transformer-based NMT.

Moreover, the pretrained language model (LM) [⁵] could be used to improve low-resource NMT. We have used the Marian^fn toolkit that allows to use the pretrained language model (LM) in the training process of NMT.

We have utilized the monolingual data of the target language to train and generate an LM using the transformer model, and the weight matrices are loaded from the pretrained LM by initializing the decoder of an encoder-decoder architecture of transformer-based NMT. We have named AsLM for the custom pretrained Assamese LM.

5.3 Post-processing

The post-processing step is used to handle out-of-vocabulary issue. It arises due to the named-entities, compounds, technical terms and misspelled words [²]. The OOV is of two types: Completely Out-of-Vocabulary (COOV) and Sense Out-of-Vocabulary (SOOV).

If the words are not present in the training data, then it is known as COOV, on the other hand SOOV are those words which are present in the training data with different usage or sense from the test set words. NMT generates <unk> (unknown) tokens against OOV.

Furthermore, NMT shows weakness in case of rare word translation since fixed-size vocabulary, which forces producing <unk> [²⁷]. The authors [⁴⁰] introduced byte pair encoding (BPE) to handle the OOV issue. Likewise, we have used BPE and proposed to use a post-processing step.

The post-processing step contains two key components: Bilingual Dictionary and Transliteration Module Bilingual Dictionary: We have prepared a bilingual English - Assamese dictionary since there is lack of available dictionary data for En-As pair.

In our previous work [²²], we have collected 200,151 a number of En-As parallel sentences from an online dictionary, namely, Glosbe.

Moreover, we have extracted 10, 103, 84 phrase pairs from the train set (as discussed in Section 5.1). We have used both (Glosbe and phrase pairs) to filter out single and double parallel words.

In the prepared dictionary, the total number of parallel single/double words are 464,586, wherein 87,024 from Glosbe and rest are from phrase pairs. We have filtered parallel noun phrases from the phrase pairs using two steps: first, extracted noun phrases from the English side of phrase pairs using NLTK^fn tool and then mapped those sentences in the phrase pairs to collect corresponding Assamese noun phrases.

The bilingual dictionary is used to replace the <unk> tokens with the appropriate target words concerning source words. Transliteration Module: We have used this module to source words which are not present in the bilingual dictionary.

It is mainly used to handle the unseen tokens that produce <unk>. We have used indic-trans^fn [⁶] to convert the source word into the target word script in the predicted sentence for both En-to-As and As-to-En transliteration.

6.1 Experimental Setup

We have employed two setups in the baseline system experiments, namely, phrase-based SMT (PBSMT) and NMT. For PBSMT, the Moses^fn [¹⁸] toolkit is used, wherein, GIZA++ [³¹] and IRSTLM [⁹] are used to extract phrase pairs to build the translational model and language model, following default settings of Moses.

The NMT experiments are carried out using the publicly available Marian [¹³] toolkit in three basic operations, data preprocessing, training and testing.

In the data preprocessing step, the word-segmentation technique, namely, byte pair encoding (BPE) [⁴⁰] with 32k merge operations is utilized. The vocabulary size of English and Assamese are 32,404 and, 31,920 at sub-word level (BPE).

Moreover, we have used GloVe [³⁵] word embeddings as subword level, wherein, the pretraining is performed up to 100 iterations with embedding vector size 200. We have named AsGloVe for custom Assamese word embeddings on Assamese side monolingual data.

For RNN-based NMT, we have investigated RNN and bidirectional RNN in our previous work [²³, ²⁴] and it is observed that the bidirectional RNN-based NMT shows better translational accuracy.

Thus, we have considered bidirectional RNN-based NMT in baseline-2, where, 0.3 drop-out in two-layer LSTM-based encoder-decoder architecture is used [²⁶].

The default configuration of six layers with eight attention heads, drop-out of 0.1, and Adam optimizer with a learning rate of 0.001 are used in the training process of NMT and LM.

A single NVIDIA Quadro P2000 GPU is utilized to train the models with early stopping criteria, i.e., the model training is halted if it does not converge on the validation set for more than 10 epochs.

6.2 Result and Error Analysis

We have used automatic evaluation metrics, namely, BLEU, TER, RIBES, METEOR, F-measure, and human evaluation scores to evaluate the quantitative results of predicted translations.

Table 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, reports the comparative BLEU score results of exploring the transformer-based NMT in different configurations i.e, with or without domain-specific parallel data, phrase pairs augmentation, synthetic parallel data, prior alignment, pretrained LM and along with the post-processing step on test set-1 and test set-2.

Furthermore, we have reported statistical significance in Table 19, wherein, BLEU scores are evaluated on the test set-1 in four groups of sentence length. It is noticed that translation accuracy decreases as the increase in sentence length (number of words).

The effect of LM is realized in the sentences of group 4 (length: 46 - 80). Table 20 and 21 presents comparative results in terms of different automatic evaluation metrics of our best model (enhanced transformer-based NMT) over the baseline transformer model (baseline-3).

Figure 2 presents comparative results of our best model over the existing works [²³, ²⁴] in terms of BLEU scores. All facets of translation accuracy cannot be evaluated using the automatic evaluation measures. Thus, the human evaluation (HE) or manual evaluation metric is taken into account. It consists of two aspects: adequacy and fluency.

The adequacy factor measures how well the predicted translation, which corresponds to the reference sentence, is contextually represented. Whereas, fluency is a different criterion that determines whether the predicted translation is well-formed or not.

The overall rating^fn of HE is calculated by the average score of adequacy and fluency. For example, if a reference sentence is: “He is coming to the park” and the predicted sentence is: “He is a good boy.” Here, the predicted sentence, inadequate with respect to the reference sentence.

But, the predicted sentence is fluent since it is a well-formed or grammatical well-structured sentence. We have hired three human evaluators who possess linguistic knowledge of both the languages, i.e., English and Assamese, and considered the assessment criteria on a scale of 1-5 on randomly selected 100 sample sentences following [³³]. Table 22, 23 report the manual evaluation results of transformer model (baseline) and the best model, wherein, the average scores of three human evaluators are presented.

From the quantitative results, it is observed that our best model (M19) attains higher translation accuracy than the baseline models.

Also, it is observed that As-to-En translation attains higher translational performance than En-to-As.

It is because of the presence of more number of tokens in En side as compared to As (as mentioned in Table 2) and as a result, more number of En tokens are encoded by the encoder and the decoder can produce a better translation for As-to-En translation.

From Table 8, it is observed the NMT performance lowers in M1 (without domain-specific parallel train set) [¹⁹], therefore, by contributing domain-specific parallel corpus in this work, NMT translational performance improves for both directions of translation covering various domains.

To closely analyse the effect of domain-specific parallel data, the sample predicted sentences of best model and with Google^fn and Bing^fn translation are discussed using the following notations:

1. (a) Example-1 (Agriculture): En-to-As

SS: More than 50 percent of these bamboos are found across the North East including Assam.

TT: (yare 50 shatangshu adhik banh asomake dhari samagra uttar purtwanchalat poua jai)

PT1: (praiy 50tatki adhik banhhbilak dhari uttarapurbanchalat poua jai)

BT: (yare 50 shatanshtkio adhik banh asamake dhari uttar-purbanchalat poua jai)

GT: (asamake dhari samagr uttar purbanchalat 50 shatanshtaki adhik banh poua jai)

1. (b) Example-1 (Agriculture): As-to-En

SS (yare 50 shatangshu adhik banh asomake dhari samagra uttar purtwanchalat poua jai)

TT: More than 50 percent of these bamboos are found across the North East including Assam.

PT2: More than 50 per cent bamboo available in the state of North East India.

BT: More than 50 per cent of these bamboos are found in the entire north-eastern region including Assam.

GT: More than 50 per cent of this bamboo is found in the entire North East including Assam.

Discussion: In the above examples, both directions of predicted translation of PT1 and PT2 are fluent like BT and GT. However, predicted translations are partial adequate unlike BT and GT, since PT2 misses “including Assamese”, and plural form of “bamboo”. Whereas, PT1 misses “”

2. (a) Example-2 (Social Media): En-to-As

SS: The moon hangs in the sky like a huge plate.

TT: (akashat prakando thalikhanar darei jonvaijani ulomi aache.)

PT1: (akashat ulomi thaka chandra)

BT: (chandrato eta dangor plater dore aakasot ulomi ase)

GT: (vishal plator dore akashat ulomi ase chandra)

2. (b) Example-2 (Social Media): As-to-En

SS: (akashat prakando thalikhanar darei jonvaijani ulomi aache.)

TT: The moon hangs in the sky like a huge plate.

PT2: Jonbil is hanging in the sky.

BT: The zonbaijani is hanging in the sky like a huge thali.

GT: The moon hangs in the sky like a huge plate.

Discussion: Here, both PT1 and PT2 generate partially adequate translation, but sentences are fluent like BT and GT. Also, unlike GT, BT unable to produce correct word (“zonbaijani”, “thali” ) for As-to-En translation.

3. (a) Example-3 (Judiciary): En-to-As

SS: The respondent asserted that after show cause notice dated 15th June 2001 was replied by the petitioner by letter dated 8th July.

TT:

PT1:

BT:

GT:

3. (b) Example-3 (Judiciary): As-to-En

SS:

TT: The respondent asserted that after show cause notice dated 15th June 2001 was replied by the petitioner by letter dated 8th July.

PT2: The respondent asserted that after the issuance of the show cause notice dated 15 June 2001 the petitioner submitted its reply by the letter dated 8th July.

BT: The respondent asserted that after the show cause notice dated June 15, 2001, the petitioner had replied to it by letter dated July 8.

GT: The respondent asserted that after the show cause notice dated 15 June 2001, the petitioner replied to it by letter dated 8 July.

Discussion: In the above examples, PT1 and PT2 show weakness in adequacy since both unable to produce correct translation of last sub-phrase “by the petitioner by letter dated 8th July” / unlike BT and GT. However, fluency is fine in all the predicted translations.

4. (a) Example-4 (Government Office): En-to-As

SS: Official receiver or assignee in insolvency proceedings

TT:

PT1:

BT:

GT:

4. (b) Example-4 (Government Office): As-to-En

SS:

TT: Official receiver or assignee in insolvency proceedings.

PT2: Official resource in bankruptcy proceeding or the allocation.

BT: Official receiver or allottee in insolvency proceedings.

GT: The official receiver or allocator in bankruptcy proceedings.

Discussion: Here, PT1 and PT2 produce inadequate as well as not fluent translations, unlike BT ang GT.

5. (a) Example-5 (Tourism): En-to-As

SS: Taj Mahal ticket to increase by Rs 200.

TT:

PT1:

BT:

GT:

5. (b) Example-5 (Tourism): As-to-En

SS:

TT: Taj Mahal ticket to increase by Rs 200.

PT2: 200 Rs will increase in Taj Mahal.

BT: Taj Mahal tickets to be increased by Rs 200.

GT: Tickets for the Taj Mahal will be increased by Rs.

Discussion: Here, PT2 missed the word “ticket”, that leads to inadequate translation Unlike BT. Whereas, GT unable produce “200” in output. However, PT1 produce correct translation like BT and GT in terms of both adequacy and fluency factors of translation.

6. (a) Example-6 (COVID-19): En-to-As

SS: The fresh order comes amid concerns in the government about the Covid19 lockdown disrupting the supply chain of essential goods.

TT:

PT1:

BT:

GT:

6. (b) Example-6 (COVID-19): As-to-En

SS:

TT: The fresh order comes amid concerns in the government about the Covid19 lockdown disrupting the supply chain of essential goods.

PT2: The new orders have come when the Covid19 lockdown avoids an essential commotion.

BT: The new order comes amid concerns in the government about the Covid-19 lockdown disrupting the supply chain of essential commodities.

GT: The new directive comes amid concerns in the government that the lockdown has disrupted the supply chain of essential commodities.

Discussion: Both PT1 and PT2 yield fluent translation like BT and GT. But partially adequate translation in PT1 and PT2, unlike BT and GT.

7. (a) Example-7 (Sports): En-to-As

SS: Indian boxers to start practice for Olympics from June 10.

TT:

PT1:

BT:

GT:

7. (b) Example-7 (Sports): As-to-En

SS:

TT: Indian boxers to start practice for Olympics from June 10.

PT2: Indian boxers should start on Olympics from June 10.

BT: Indian boxers to start training for Olympics from June 10.

GT: Indian boxers will start training for the Olympics from June.

Discussion: Both PT1 and PT2 missed the word “practice” or in output, that lead to partially adequate unlike GT and BT. However, all the sentences are fluent.

8. (a) Example-8 (Literature): En-to-As

SS: A practice called Mizwah has been prevalent among Jewish people.

TT:

PT1:

BT:

GT:

8. (b) Example-8 (Literature): As-to-En

SS:

TT: A practice called Mizwah has been prevalent among Jewish people.

PT2: A practice of worship is prevalent among Jewish people.

BT: There has been a custom called Mizwah among the Jewish people.

GT: There is a custom called mizvah among the Jewish people.

Discussion: Like BT and GT, both PT1 and PT2 generate fluent translation. However, inadequate translation in case of PT1 and PT2 unlike BT and GT.