Backpropagation through Time Algorithm for Training Recurrent Neural Networks using Variable Length Instances

Algoritmo de retropropagación a través de tiempo para el aprendizaje de redes neuronales recurrentes usando instancias de longitud variable

Isel Grau¹, Gonzalo Nápoles¹, Isis Bonet², and María Matilde García¹

¹ Centro de Estudios de Informática, Universidad Central "Marta Abreu" de Las Villas, Cuba.

Abstract

Artificial Neural Networks (ANNs) are grouped within connectionist techniques of Artificial Intelligence. In particular, Recurrent Neural Networks are a type of ANN which is widely used in signal reproduction tasks and sequence analysis, where causal relationships in time and space take place. On the other hand, in many problems of science and engineering, signals or sequences under analysis do not always have the same length, making it difficult to select a computational technique for information processing. This article presents a flexible implementation of Recurrent Neural Networks which allows designing the desired topology based on specific application problems. Furthermore, the proposed model is capable of learning to use knowledge bases with instances of variable length in an efficient manner. The performance of the suggested implementation is evaluated through a study case of bioinformatics sequence classification. We also mention its application in obtaining artificial earthquakes from seismic scenarios similar to Cuba.

Keywords: Recurrent neural networks, backpropagation through time, sequence analysis, bioinformatics, artificial earthquakes.

Resumen

Las Redes Neuronales Artificiales (RNAs) se agrupan dentro de las técnicas conexionistas de la Inteligencia Artificial. En particular las Redes Neuronales Recurrentes son un tipo de RNA de amplio uso en tareas de reproducción de señales y análisis de secuencias, donde se reflejan relaciones causales en el tiempo y el espacio respectivamente. Por otra parte, en muchos problemas de la ingeniería y la ciencia, las señales o secuencias analizadas no siempre tienen la misma longitud, dificultando la selección de la técnica computacional a utilizar para su procesamiento. En este artículo se presenta una implementación flexible de Redes Neuronales Recurrentes que permite definir la topología deseada en función del problema específico de aplicación. Además este modelo es capaz de aprender utilizando bases de conocimiento con instancias de longitud variable de una forma eficiente. El rendimiento de la implementación propuesta es evaluado a través de un caso de estudio de clasificación de secuencias bioinformáticas y además se describe su aplicación en la obtención de terremotos sintéticos a partir de información de escenarios sísmicos similares a los de Cuba.

Palabras clave: Redes neuronales recurrentes, retropropagación a través de tiempo, análisis de secuencias, bioinformática, terremotos artificiales.

References

1. García, M.M., Bello, R., Díaz, A. & Reynoso, A. (2003). Redes Neuronales Artificiales. México: Prometeo Editores. Retrieved from http://148.202.105.23:8991/F/8TGDD3KRGKKHD2X8L9M8EHYA9HKBS7CSJHT9FE5Y4BKDGM2 UUT-01381?func=item-global&doc_library=UDG01&doc_number=000219225&year=&volume=&sub_library=EIC.         [ Links ]

2. Hammer, B. & Villmann, T. (2003). Mathematical Aspects of Neural Networks. 11^th European Symposium on Artificial Neural Networks (ESANN 2003), Bruges, Belgium, 59-72.         [ Links ]

3. Pearlmutter, B.A. (1990). Dynamic Recurrent Neural Networks (CMU-CS-90-196). Pittsburgh, PA: Carnegie Mellon University.         [ Links ]

4. Baldi, P., Pollastri, G., Frasconi, P., & Vullo, A. (2003). New Machine Learning Methods for the Prediction of Protein Topologies. In Paolo Frasconi & Ron Shamir (Eds.). Artificial Intelligence and Heuristic Methods in Bioinformatics (51-74). Amsterdam; Washington, DC: IOS Press.         [ Links ]

5. Draghici, S. & Potter, R.B. (2003). Predicting HIV drug resistance with neural networks. Bioinformatics, 19(1), 98-107.         [ Links ]

6. Bonet, I., Garcia, M.M., Saeys, Y., Peer, Y., & Grau, R. (2007). Predicting Human Immunodeficiency Virus (HIV) Drug Resistance Using Recurrent Neural Networks. 2^nd International Work-Conference on The Interplay Between Natural and Artificial Computation (IWINAC'07). Lecture Notes in Computer Science, 4527, 234-243.         [ Links ]

7. Beerenwinkel, N., Lengauer, T., Selbig, J., Schmidt, B., Walter, H., Korn, K., Kaiser, R., & Hoffmann, D. (2001). Geno2pheno: interpreting genotypic HIV drug resistance tests. IEEE Intelligence System, 16(6), 35-41.         [ Links ]

8. Beerenwinkel, N., Schmidt, B., Walter, H., Kaiser, R., Lengauer, T., Hoffmann, D., Korn, K., & Selbig, J. (2002). Diversity and complexity of HIV-1 drug resistance: a bioinformatics approach to predicting phenotype from genotype. Proceedings of the National Academy of Sciences of the United States of America, 99(12), 8271 -8276.         [ Links ]

9. Bonet, I., Rodríguez, A., Grau, R., García, M.M., & Izquierdo, Y. (2008). Predicting HIV protease drugs resistance with multiclassifier systems. XVII International AIDS Conference, México City.         [ Links ]

10. Bonet, I., Rodríguez, A., García, M.M., & Grau, R. (2012). Combinación de clasificadores para bioinformática. Computación y Sistemas, 16(2), 191 -201.         [ Links ]

11. Grau, I., Nápoles, G., León, M., & Grau, R. (2012). Fuzzy Cognitive maps for modelling, predicting and interpreting HIV drug resistance. Lecture Notes in Computer Science, 7637, 31-40.         [ Links ]

12. Nápoles, G., Grau, I., León, M. & Grau, R. (2012). Modelling, aggregation and simulation of a dynamic biological system through Fuzzy Cognitive Maps. 11th Mexican International Conference on Artificial Intelligence, (MICAI 2012) Part II. Lecture Notes in Artificial Intelligence, 7630, 187-197.         [ Links ]

13. Rhee, S., Taylor, J., Wadhera, G., Ben-Hur, A., Brutlag, D.L., & Shafer, R.W. (2006). Genotypic predictors of human immunodeficiency virus type 1 drug resistance. Proceedings of the National Academic of Sciences of the United States of America, 103(46), 17355-17360.         [ Links ]

14. Kierczak, M., Ginalski, K., Dramiriski, M., Koronacki, J., Rudnicki, W., & Komorowski, J.(2009). A Rough Set-Based Model of HIV-1Reverse Transcriptase Resistome. Bioinformatics and Biology Insights, 5(3), 109-127.         [ Links ]

15. Vermeiren, H., Van den Bulcke, T., Van Marck, H., Lecocq, P., Van Houtte, M., & Bacheler, L. (2004). Application of multiple linear regression modelling to the quantitative prediction of HIV-1 drug susceptibility phenotype from viral genotype. XIII International HIV Drug Resistance Workshop, Costa Adeje, Canary Islands, Spain.         [ Links ]

16. Hilera, J.R. & Martinez, V.J. (1995). Redes Neuronales Artificiales: Fundamentos, Modelos y Aplicaciones., Madrid: RA-MA.         [ Links ]

17. Bonet, I., Salazar, S., Rodríguez, A., Grau, R., & García, M.M. (2007). Redes Neuronales Recurrentes para Análisis de Secuencias. Revista Cubana de Ciencias Informáticas, 1(4), 48-57.         [ Links ]

18. Atiya, A.F. & Parlos, A.G. (2000). New Results on Recurrent Network Training: Unifying the Algorithms and Accelerating Convergence. IEEE Transactions on Neural Networks, 11(3), 697-709.         [ Links ]

19. Werbos, P.J. (1990). Backpropagation Through Time: What it does and How to do it. Proceedings of the IEEE, 78(10), 1550-1560.         [ Links ]

20. Prosperi, M. & Ulivi, G. (2008). Evolutionary Fuzzy Modelling for Drug Resistant HIV-1 Treatment Optimization. Studies in Computational Intelligence, 82, 251 -287.         [ Links ]

21. Beerenwinkel, N., Sing, T., Lengauer, T., Rahnenfuhrer, J., Roomp, K., Savenkov, I., Fischer, R., Hoffmann, D., Selbig, J., Korn, K., Walter, H., Berg, T., Braun, P., Fatkenheuer, G., Oette, M., Rockstroh, J., Kupfer, B., Kaiser, R., & Daume, M. (2005). Computational methods for the design of effective therapies against drug resistant HIV strains. Bioinformatics, 21(21),3943-3950.         [ Links ]

22. HIV Drug Resistance Database (2012). Retrieved from http://hivdb.stanford.edu//cgi-bin/GenoPhenoDS.cgi        [ Links ]

23. Miyazawa, S. & Jernigan, R.L. (1999). Self-Consistent Estimation of Inter-Residue Protein Contact Energies Based on an Equilibrium Mixture Approximation of Residues. Proteins: Structure, Function, and Bioinformatics, 34(1), 49-68.         [ Links ]

24. Friedman, M. (1937). The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 32(200), 675-701.         [ Links ]

25. Friedman, M. (1940). A comparison of alternative tests of significance for the problem of m rankings. Annals of Mathematical Statistics, 11(1), 86-92.         [ Links ]

26. Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin, 1(6), 80-83.         [ Links ]