nova página do texto(beta)
Inglês (pdf)
Artigo em XML
Referências do artigo
Tradução automática
Enviar este artigo por email
Citado por SciELO 
Similares em
SciELO 
Permalink
1 Introduction
Multi-modal content is gaining popularity in machine translation (MT) community due to its appealing chances to improve translation quality by considering information available only in the visual context of a sentence. Handling multi-modal content also has its usage in commercial applications such as translation of image captions in online news articles or machine translation for e-commerce product listings [11,3,9,22]. Although the general performance of neural machine translation (NMT) models is very good given large amounts of parallel texts, some inputs can remain genuinely ambiguous, especially if the input context is limited.
One example is the word "mouse" in English (source) which can be translated into different words in Hindi based on the context (e.g. either a computer mouse or a small rodent), another example is given in Fig. 1 below. There is a limited number of multi-modal datasets available and even fewer of them are also multi-lingual. Our aim is to extend the set of languages available for multi-modal experiments by adding a Hindi variant of a subset of Visual Genome.
Fig. 1 A sample image from Visual Genome with several rectangular regions annotated with English captions. The caption of region 1 was processed by baseline MT into Hindi, resulting in a translation error (underlined)
Creating such a dataset enables multi-modal experimenting with Hindi for various applications and it could also facilitate the exploration of how the language is grounded in vision.
Visual Genome (http://visualgenome.org/, [10]) is a large set of real-world images, each equipped with annotations of various regions in the image. The annotations include a plain text description of the region (usually sentence parts or short sentences, e.g. "a red ball in the air") and also several other formally captured types of information (objects, attributes, relationships, region graphs, scene graphs, and question-answer pairs). We focus only on the textual descriptions of image regions and provide their translations into Hindi.
An example of a Visual Genome image with captions is given in Fig. 1. Region 1 is one of the captions in Visual Genome ("Two lambs lying in the sun.") and we illustrate how MT into Hindi mistranslated the word "lying" as "telling lies". While such a meaning is possible (in fairy tales), the image clearly hints towards to correct meaning. In this particular case, the surrounding captions could help but no direct evidence for or against the two meanings is available in the textual information.
The main portion of our Hindi Visual Genome is intended for training purposes of tools like multi-modal translation systems or Hindi image labelers. Every item consists of an image, a rectangular region in the image, the original English caption from Visual Genome and finally our Hindi translation. Additionally, we create a challenge test set with the same structure but a different sampling that promotes the presence of ambiguous words in the English captions with respect to their meaning and thus their Hindi translation. The final corpus statistics of the "Hindi Visual Genome" are in Table 1.
Table 1 Hindi Visual Genome corpus details. One item consists of an English source segment, its Hindi translation, the image and a rectangular region in the image
| Data Set | Items |
|---|---|
| Training Set | 28,932 |
| Development Test Set (D-Test) | 998 |
| Evaluation Test Set (E-Test) | 1,595 |
| Challenge Test Set (C-Test) | 1,400 |
The paper is organized as follows: In Section 2, we survey related multi-modal multi-lingual datasets. Section 3 describes the way we selected and prepared the training set. Section 4 is devoted to the challenge test set: the method to find ambiguous words and the steps taken when constructing the test set, its final statistics and a brief discussion of our observations. We conclude in Section 5.
2 Related Work
Multi-modal neural machine translation is an emerging area where translation takes more than text as input. The additional information sources could be features from images or sound. Combining visual features with language modeling has shown better result for image captioning and question answering [14,21,12].
Many experiments were carried out considering images to improve machine translation, i.a. for resolving ambiguity due to different senses of words in different contexts. One of the starting points were multi-modal shared tasks at WMT [19,7,2] organized around "Flickr30k" [8], a multi-modal multi-lingual (English-German, English-French, and English-Czech) dataset.
[4] proposed a multi-modal NMT system using image feature for Hindi-English language pair. Due to the lack of English-Hindimulti-modal data, they used a synthetic training dataset and manually curated development and test sets for Hindiderived from the English part of Flickr30k corpus [17]. [1] proposed a probabilistic method using pictures for word prediction constrained to a narrow set of choices, such as possible word senses. Their results suggest that images can help word sense disambiguation.
Different techniques then followed, using various neural network architectures for extracting and using the contextual information. One of the approaches was proposed by [11] for multi-modal translation by replacing image embedding with an estimated posterior probability prediction for image categories.
3 Training and Test Set Preparations
To produce the main part of our corpus, we have automatically translated and manually post-edited the English captions of "Visual Genome" corpus into Hindi.
The starting point were 31,525 randomly selected images from Visual Genome. Of all the English-captioned regions available for each of the images, we randomly select one. To obtain the Hindi translation, we have followed these steps:
We translated all 31,525 captions into Hindi using the NMT model (Tensor-to-Tensor, [20]) specifically trained for this purpose as described in [16].
We uploaded the image, the source English caption and its Hindi machine translation into a "Translation Validation Website",1 which we designed as a simple interface for post-editing the translations. One important feature was the use of a Hindi on-screen keyboard2 to enable proper text input even for users with limited operating systems.
Our volunteers post-edited all the Hindi translations. The volunteers were selected based on their Hindi language proficiency.
We manually verified and finalized the post-edited files to obtain the training and test data.
To facilitate comparison of future experiments with this dataset, we create a dedicated development and test set. The split of the 31,525 items into the training, development and test sets with sizes as listed in Table 1 was random.
4 Challenge Test Set Preparations
In addition to the randomly selected 31,525 items described above, we prepared a challenge test set of 1,400 segments which are more likely to benefit from visual information for word sense disambiguation. To achieve this targeted selection, we first found the most ambiguous words from the whole "Visual Genome" corpus and then extracted segments containing these words. The overall steps for obtaining the ambiguous words are shown in Fig. 2.
The detailed sequence of processing steps was as follows:
Translate all English captions from the Visual Genome dataset (3.15 millions unique strings) using a baseline machine translation system into Hindi, obtaining a synthetic parallel corpus. In this step, we used Google Translate.
Apply word alignment on the synthetic parallel corpus using GIZA++ [15], in a wrapper3 that automatically symmetrizes two bidirectional alignments; we used the intersection alignment.
Extract all pairs of aligned words in the form of a "translation dictionary". The dictionary contains key/value pairs of the English word (E) and all its Hindi translations
Train Hindi word2vec (W2V) [13] word embeddings. We used the gensim4[18] implementation and trained it on IITB Hindi Monolingual Corpus5 which contains about 45 million Hindi sentences. Using such a large collection of Hindi texts improves the quality of the obtained embeddings.
For each English word from the translation dictionary (see Step 3), get all Hinditranslation words and their embeddings (Step 4).
-
Apply K-means clustering algorithm to the embedded Hindi words to organize them according to their word similarity.
If we followed a solid definition of word senses and if we knew how many senses there are for a given source English word and how they match the meanings of the Hindi words, the K would correspond to the number of Hindi senses that the original English word expresses. We take the pragmatic approach and apply K-means for a range of values (K from 2 to 6).
-
Evaluate the obtained clusters with the Silhouette Score, Davies-Bouldin Index (DBI), and Calinski-Harabaz Index (CHI) [6,5]. Each of the selected scores reflects in one way or another the cleanliness of the clusters, their separation. For the final sorting (Step 8), we mix these scores using a simple average function.
The rationale behind using these scores is that if the word embeddings of the Hindi translations can be clearly clustered into 2 or more senses, then the meaning distinctions are big enough to indicate that the original English word was ambiguous. The exact number of different meanings is not too important for our purpose.
Sort the list in descending order to get the most ambiguous words (as approximated by the mean of clustering measures) at the top of the list.
Manually check the list to validate that the selected ambiguous words indeed potentially need an image to disambiguate them. Select a cutoff and extract the most ambiguous English words.
The result of this semi-automatic search and manual validation of most ambiguous words was a list of 19 English words, see Table 2.
Table 2 Challenge test set: distribution of the ambiguous words
| Word | Segment Count | |
|---|---|---|
| 1 | Stand | 180 |
| 2 | Court | 179 |
| 3 | Players | 137 |
| 4 | Cross | 137 |
| 5 | Second | 117 |
| 6 | Block | 116 |
| 7 | Fast | 73 |
| 8 | Date | 56 |
| 9 | Characters | 70 |
| 10 | Stamp | 60 |
| 11 | English | 42 |
| 12 | Fair | 41 |
| 13 | Fine | 45 |
| 14 | Press | 35 |
| 15 | Forms | 44 |
| 16 | Springs | |30 |
| 17 | Models | 25 |
| 18 | Forces | 9 |
| 19 | Penalty | 4 |
| Total | 1400 |
For each of these words, we selected and extracted a number of items available in the original Visual Genome and provided the same manual validation of the Hindi translation as described in Section 3 for the training and regular test sets. We tried to make a balance and the frequencies of the ambiguous words in the challenge test set roughly correspond to the original frequencies in Visual Genome.
Incidentally, 7 images and English captions occur in both the training set and the challenge test set.6 The overlap in images (but using different regions and captions) is larger: 359.
Fig. 3 illustrates two sample items selected for the word "penalty" (Hindi translation omitted here). We see that for humans, the images are clearly disambiguating the meaning of the word: the fine to be paid for honking vs. the kick in a soccer match.
Arguably, the surrounding English words in the source segments (e.g. "street" vs. "white lined") can be used by machine translation systems to pick the correct translation even without access to the image. The size of the original dataset of images with captions however did not allow us to further limit the selection to segments where the text alone is not sufficient for the disambiguation.
5 Conclusion and Future Work
We presented a multi-modal English-to-Hindi dataset. To the best of our knowledge, this is the first such dataset that includes an Indian language. The dataset can serve e.g. in Hindi image captioning but our primary intended use case was research into the employment of images as additional input to improve machine translation quality.
To this end, we created also a dedicated challenge test set with text segments containing ambiguous words where the image can help with the disambiguation.
All parts of our dataset served in WAT 20197 shared task on multi-modal translation.8
We illustrated that the text-only information in the surrounding words could be sufficient for the disambiguation. One interesting research direction would be thus to ignore all the surrounding words and simply ask: given the image, what is the correct Hindi translation of this ambiguous English word. Another option we would like to pursue is to search larger datasets for cases where even the whole segment does not give a clear indication of the meaning of an ambiguous word.
Our "Hindi Visual Genome" is available for research and non-commercial use under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 License9 at http://hdl.handle.net/11234/1-2997.
