MIT researchers have developed a new dataset designed to help artificial intelligence models understand charts more accurately, a task that remains difficult even for advanced vision-language systems. The dataset, called ChartNet, could improve how AI tools analyze business trends, summarize financial reports and interpret scientific figures.
Charts are widely used in business, finance, research and policy reports because they combine visual design, numbers and written labels in one compact format. But that combination is exactly what makes them hard for AI models to interpret. A model may need to understand the chart type, read the labels, compare values, identify patterns and explain the meaning in language. Even strong commercial systems can still produce incomplete or incorrect answers when asked to summarize charts.
What ChartNet Is Designed to Solve
Researchers from MIT and the MIT-IBM Computing Research Lab created ChartNet to address a major training gap. Many existing datasets for chart understanding are too small, too limited or pulled from the internet without enough supporting information. That makes it harder for vision-language models to learn how charts actually work and how visual elements connect to numerical data.
ChartNet includes more than one million chart images, along with the code used to generate each chart, written descriptions, numerical tables and question-and-answer pairs. This gives AI models multiple ways to connect the same information: the visual chart, the underlying numbers and the language used to describe it. As a result, the model can learn not only to recognize a chart, but also to reason about what the chart is showing.
Why Chart Understanding Matters
Chart interpretation is especially important in industries such as finance, consulting, research and business intelligence. Companies often rely on charts to show revenue trends, market movement, customer behavior, risk levels or operational performance. If an AI system can accurately extract and summarize chart information, it could help teams process reports faster and make decisions with less manual work.
The same applies to science. Researchers frequently use figures to present experimental results, comparisons and trends. Better AI chart understanding could help scientists review papers, compare results and extract useful insights from complex visual data more efficiently.
How the Dataset Was Built
To build ChartNet, the researchers used a two-step synthetic data generation process. First, their system translated existing chart examples into code. Then it modified that code in many ways, changing chart types, colors, values, topics and other visual elements. Starting from one chart, the system could create many different versions, allowing the team to build a much larger and more diverse dataset.
The team also added automated quality checks to ensure that the generated charts were clean, accurate and meaningful. This matters because synthetic data is only useful if it teaches models the right patterns. ChartNet also includes some data points reviewed by human experts, giving the dataset an extra layer of reliability for more complex chart examples.
Smaller Open-Source Models Performed Better
The researchers used ChartNet to train several open-source vision-language models, including IBM’s Granite Vision series. After training, many of these smaller models outperformed much larger commercial models on tasks such as chart data extraction, chart summarization, chart reconstruction and chart question answering.
That result is important because it suggests that better training data can sometimes matter more than simply using a larger model. If smaller open-source models can understand charts more accurately, businesses and researchers with limited budgets may be able to use powerful AI tools without relying only on expensive commercial systems.
Final Outlook
ChartNet could become an important step toward making AI models better at reading the charts that appear in business reports, scientific papers and financial analysis. Instead of treating charts as simple images, the dataset teaches models to connect visuals, numbers and language in a more complete way.
The work also shows a broader trend in artificial intelligence: improving performance is not only about building bigger models, but also about giving models better, richer and more carefully designed training data. If ChartNet continues to expand, it could help make chart analysis faster, more accurate and more accessible across many industries.