Analysis of Text-to-SQL Benchmarks: Limitations, Challenges and Opportunities Anna MitsopoulouAthena Research Center, Greeceanna.mitsopoulou@athenarc.gr Georgia KoutrikaAthena Research Center, Greecegeorgia@athenarc.gr ABSTRACT Despite being a fast-paced research field, text-to-SQL systems face critical challenges. The datasets used for the training and evaluation of these systems play a vital role in determining their performance as well as the progress in the field. In this work, we introduce a methodology for text-to-SQL dataset analysis, and we perform an in-depth analysis of several text-to-SQL datasets, providing valuable insights into their capabilities and limitations and how they affect training and evaluation of text-to-SQL systems. We investigate existing evaluation methods, and propose an informative system evaluation based on error analysis. We show how our dataset analysis can help explain the behavior of a system on different datasets. Using our error analysis, we further show how we can pinpoint the sources of errors of a text- to-SQL system for a particular dataset and reveal opportunities for system improvements. 1 INTRODUCTION Text-to-SQL systems translate natural language (NL) questions to SQL relieving users from the use of SQL for accessing data in relational databases. In recent years, text-to-SQL systems [37 , 44, 51 , 58 ] have achieved significant advancements due to the use of large language models (BERT [ 9 ], T5 [ 49 ], GPT [ 48]) and the creation of task-specific datasets (e.g., WikiSQL [ 75 ], Spider [ 68 ]) used for system training and evaluation. These approaches tackle the text-to-SQL problem as a language translation prob-lem, and they train a neural network on a large amount of {NL question/SQL query} pairs [27].Unfortunately, unlike systems that translate from one natural language to another, or from natural language to code (e.g.,Python), text-to-SQL systems face challenges and do not enjoy asbroad adoption, despite the high competition that exists among them. The datasets used for the training and evaluation of text-to SQL systems play a vital role in the performance of these systems as well as in determining the progress in the field. While a system trained on a benchmark like Spider [68] may exhibit good performance on this benchmark, when it is used on a different benchmark or used in a real application/domain,it does not fare as well. Several factors, such as the type of SQL queries, their distribution, the domains, and even the size of the data, matter when training a system. A system cannot perform well for unseen (or even not “seen enough”) queries or data. On the other hand, when evaluating a text-to-SQL system, a text to-SQL benchmark may create false expectations on the query translation capabilities of the system. For example, a system achieving 80% accuracy on a dataset with simple queries could be worse than one achieving 60% accuracy on a dataset with more complex queries. An absolute accuracy number does not© 2025 Copyright held by the owner/author(s). Published in Proceedings of the 28th International Conference on Extending Database Technology (EDBT), 25th March-28th March, 2025, ISBN 978-3-98318-097-4 on OpenProceedings.org. Distribution of this paper is permitted under the terms of the Creative Commons license CC-by-nc-nd 4.0. provide much insight unless we consider the characteristics of the evaluation dataset, such as the types and distributions of NL and SQL queries. It is also important to be able to pinpoint the sources of the errors a text-to-SQL system makes, and hencereveal opportunities for system improvements. While WikiSQL [75 ] and Spider [ 68 ] are the first large-scale, multi-domain benchmarks for training and evaluating neural text-to-SQL systems, several datasets have preceded and followed them (e.g., [ 6, 15 , 32 , 38 , 70]) that serve different purposes and focus on different aspects of the text-to-SQL problem. In contrast to the effort to understand text-to-SQL systems through studies and surveys [1, 2, 7, 18, 24 , 28 , 39 , 47], extensive studies and evaluations of text-to-SQL datasets are missing. However, as we explained above, understanding the capabilities and limitations of text-to-SQL datasets is vital for making progress in the field. In this work, we present a structured survey of text-to-SQL datasets, their design objectives as well as their shortcomings. Moreover, we present a text-to-SQL dataset analysis methodology that provides a set of dimensions and measures to analyze and characterize the richness and distributions of the SQL queries, the natural language questions and the databases covered by a dataset. Using our methodology, we evaluate several datasets, and provide valuable insights into their value, complexity, and limittions. This analysis also provides insights into the limitations of current text-to-SQL systems, and reveals several opportunities for research for the development of more effective benchmarks. Furthermore, we investigate the methods and metrics for evaluating text-to-SQL systems and we point out their shortcomings. We propose an alternative in the direction of a more informative evaluation that combines a new metric and error analysis based on an automatically generated SQL query categorization that canprovide insights about the system capabilities and pain points. We show the potential of our approach for providing a more fine-grained system evaluation. In particular, we experimentally show how our dataset analysis can help explain the behavior of a system on different datasets. Using our proposed error analysis, we further show how we can pinpoint the sources of errors of a text-to-SQL system for a particular dataset.In a nutshell, our contributions are summarized as follows: We present a structured study of text-to-SQL datasets. We present a methodology for evaluating text-to-SQL datasets. We present an in-depth analysis of several text-to-SQL datasets. We provide an error analysis method combining a new metricand an automatically generated SQL query categorization. We show the potential of our dataset analysis methodologyand error analysis for more fine-grained and insightful systemevaluations. 2 TEXT-TO-SQL SYSTEMS Research on text-to-SQL systems dates back to seventies, however, recently, the use of deep learning techniques has given great boost in the development of such systems [ 31 ]. The most successful text-to-SQL systems [ 36 , 37 , 44 , 51 ] rely on pretrained language models (e.g., T5 or GPT-based architectures) and