Simplifying Encoder-Decoder-Based Neural Machine Translation Systems to Translate between Related Languages

Abstract

Neural machine translation is one of the most advanced approaches to machine translation and one that is recently obtaining good enough results to make use of it in real-life scenarios. The currently widely used architecture is what is known as sequence-to-sequence architecture with attention mechanism, which uses an encoder to create a vector representation of the input sentence in source language, a decoder to output a sentence in target language and an attention mechanism to help the decoder produce more accurate outputs. The simplification of state-of-the-art sequence-to-sequence neural machine translation with attention is explored in this work for the translation between related languages. First, some of the state-of-the-art features present in the baseline system are presented and described. The main hypothesis of this work is the possibility of removing these features without worsening the translation quality too much and simplifying the network's structure at the same time when translating between related languages. The main part of this work is the substitution of state-of-the-art attention mechanisms, used to help the decoder know which part of the source sentence is more relevant for the part of the target sentence being outputted, by a simplified attention mechanism which mostly pays attention to the word in the source sentence in the same position as the current target word. The simplification is carried out by removing beam search (a technique used to explore a wider range of possible outputs instead of limiting the output to the highest probability of being the correct output), substituting the bidirectional encoder by a unidirectional encoder and creating a new \local attention" mechanism in replacement for the current more complex state-of-the-art attention mechanism. Once the simplifications have been discussed and implemented, their impact on translation quality for related languages (Spanish and Catalan in the case of this work) is tested and compared to determine their suitability. From the results obtained, as expected, the removal of beam search and the substitution of the bidirectional encoder by a unidirectional encoder does not have a great impact on translation quality, resulting in a decrease of 6%-23% in BLEU score depending on the attention mechanism being used. On top of this, the introduction of the newly-developed \local attention" mechanism improves translation quality by 176% and 218% in BLEU score when compared to an attention-less system, about 22%-27% less than the state-of-the-art attention mechanism used in the baseline system. All of this resulting in the great simplification in the network, reducing the number of trainable parameters from 12.195.945 to 9.816.485 (19.5%) and the training time from 22h 53m to 12h 15m.