IEEE Access · 2019 · 37 citations · 35 references
ComputingComputer ScienceComputational LinguisticsComputer VisionNatural Language ProcessingDeep LearningLanguage ModelingLanguage ModelsNatural Language UnderstandingAttentionDeep Image FeaturesImage CaptioningLong Short-term Memory NetworksSalient Feature ExtractionSalient Feature IdentificationSalient Region DetectionSiamese Networks
Several deep learning techniques have been intensively reviewed for captioning tasks, enabling the possibility of textual understanding, and description of both simple and complex images. In advancing this knowledge, this paper proposes a multimodal end-to-end siamese difference captioning model (SDCM) to automatically generate a natural language description of differences in an image pair. The proposed supervised learning model combines several deep learning techniques in exploring the practicability of capturing, aligning, and computing the disparities between two image features, for the purpose of creating corresponding language model probability distribution. First, a deep siamese convolutional neural network is used to extract the feature vector discrepancies of an image pair, and then an attention mechanism enables the detection of salient regions of the feature vector which effectively allows a bidirectional long short-term memory decoder to generate a matching and semantically associated textual sequence. The evaluation of the model is tested on the spot-the-diff baseline dataset which consists of pairs of images and their equivalent captions. The results indicate that our proposed model demonstrates a highly competitive performance in comparison to the state of the art.
35
Sepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 98.1K citations
Scikit-learn: Machine Learning in Python
Fabián Pedregosa, Gaël Varoquaux, Alexandre Gramfort et al. · arXiv (Cornell University) · 2012 · 63.7K citations · Full text
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio et al. · Proceedings of the IEEE · 1998 · 58.2K citations · Full text
ImageNet Large Scale Visual Recognition Challenge
Olga Russakovsky, Jia Deng, Hao Su et al. · International Journal of Computer Vision · 2015 · 40.2K citations