What Will You Tell Me About the Chart? – Automated Description of Charts

An automatic chart description is a very challenging task. There are many more relationships between objects in a chart compared to general computer vision problems. Furthermore, charts have a different specificity to natural-scene pictures, so commonly used methods do not perform well. To tackle these problems, we propose a process consisting of three sub-tasks: (1) chart classification, (2) detection of a chart’s essential elements, and (3) generation of text description.Due to the lack of plot datasets dedicated to the task of generating text, we prepared a new dataset – ChaTa+ which contains real-made figures. Additionally, we have adjusted publicly available FigureQA and PlotQA datasets to our particular tasks and tested our method on them. We compared our results with those of the Adobe team [3], which we treated as a benchmark. Finally, we obtained comparable results of the models’ performance, although we trained them on a more complex dataset (semi-synthetic PlotQA) and built a less resource-intensive infrastructure.