Training an End-to-End System for Handwritten Mathematical Expression Recognition by Generated Patterns

Abstract

Motivated by recent successes in neural machine translation and image caption generation, we present an end-to-end system to recognize Online Handwritten Mathematical Expressions (OHMEs). Our system has three parts: a convolution neural network for feature extraction, a bidirectional LSTM for encoding extracted features, and an LSTM and an attention model for generating target LaTex. For recognizing complex structures, our system needs large data for training. We propose local and global distortion models for generating OHMEs from the CROHME database. We evaluate the end-to-end system on the CROHME database and the generated databases. The experiential results show that the end-to-end system achieves 28.09% and 35.19% recognition rates on CROHME without and with the generated data, respectively.