Please use this identifier to cite or link to this item:
https://doi.org/10.21256/zhaw-30173
Publication type: | Conference paper |
Type of review: | Peer review (publication) |
Title: | ScienceBenchmark : a complex real-world benchmark for evaluating natural language to SQL systems |
Authors: | Zhang, Yi Deriu, Jan Milan Katsogiannis-Meimarakis, George Kosten, Catherine Koutrika, Georgia Stockinger, Kurt |
et. al: | No |
DOI: | 10.14778/3636218.3636225 10.21256/zhaw-30173 |
Proceedings: | Proceedings of the VLDB Endowment |
Volume(Issue): | 17 |
Issue: | 4 |
Page(s): | 685 |
Pages to: | 698 |
Conference details: | 50th International Conference on Very Large Data Bases, Guangzhou, China, 25-29 August 2024 |
Issue Date: | Mar-2024 |
Publisher / Ed. Institution: | Association for Computing Machinery |
ISSN: | 2150-8097 |
Language: | English |
Subjects: | Database system; Latural language processing; Machine learning; Large language model |
Subject (DDC): | 005: Computer programming, programs and data 006: Special computer methods |
Abstract: | Natural Language to SQL systems (NL-to-SQL) have recently shown improved accuracy (exceeding 80%) for natural language to SQL query translation due to the emergence of transformer-based language models, and the popularity of the Spider benchmark. However, Spider mainly contains simple databases with few tables, columns, and entries, which do not reflect a realistic setting. Moreover, complex real-world databases with domain-specific content have little to no training data available in the form of NL/SQL-pairs leading to poor performance of existing NL-to-SQL systems. In this paper, we introduce ScienceBenchmark, a new complex NL-to-SQL benchmark for three real-world, highly domain-specific databases. For this new benchmark, SQL experts and domain experts created high-quality NL/SQL-pairs for each domain. To garner more data, we extended the small amount of human-generated data with synthetic data generated using GPT-3. We show that our benchmark is highly challenging, as the top performing systems on Spider achieve a very low performance on our benchmark. Thus, the challenge is many-fold: creating NL-to-SQL systems for highly complex domains with a small amount of hand-made training data augmented with synthetic data. To our knowledge, ScienceBenchmark is the first NL-to-SQL benchmark designed with complex real-world scientific databases, containing challenging training and test data carefully validated by domain experts. |
URI: | https://digitalcollection.zhaw.ch/handle/11475/30173 |
Fulltext version: | Published version |
License (according to publishing contract): | CC BY-NC-ND 4.0: Attribution - Non commercial - No derivatives 4.0 International |
Departement: | School of Engineering |
Organisational Unit: | Centre for Artificial Intelligence (CAI) Institute of Computer Science (InIT) |
Published as part of the ZHAW project: | INODE – Intelligent Open Data Exploration (EU Horizon 2020) |
Appears in collections: | Publikationen School of Engineering |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
2024_Zhang-etal_ScienceBenchmark-PVLDB2024.pdf | 608.6 kB | Adobe PDF | View/Open |
Show full item record
Zhang, Y., Deriu, J. M., Katsogiannis-Meimarakis, G., Kosten, C., Koutrika, G., & Stockinger, K. (2024). ScienceBenchmark : a complex real-world benchmark for evaluating natural language to SQL systems [Conference paper]. Proceedings of the VLDB Endowment, 17(4), 685–698. https://doi.org/10.14778/3636218.3636225
Zhang, Y. et al. (2024) ‘ScienceBenchmark : a complex real-world benchmark for evaluating natural language to SQL systems’, in Proceedings of the VLDB Endowment. Association for Computing Machinery, pp. 685–698. Available at: https://doi.org/10.14778/3636218.3636225.
Y. Zhang, J. M. Deriu, G. Katsogiannis-Meimarakis, C. Kosten, G. Koutrika, and K. Stockinger, “ScienceBenchmark : a complex real-world benchmark for evaluating natural language to SQL systems,” in Proceedings of the VLDB Endowment, Mar. 2024, vol. 17, no. 4, pp. 685–698. doi: 10.14778/3636218.3636225.
ZHANG, Yi, Jan Milan DERIU, George KATSOGIANNIS-MEIMARAKIS, Catherine KOSTEN, Georgia KOUTRIKA und Kurt STOCKINGER, 2024. ScienceBenchmark : a complex real-world benchmark for evaluating natural language to SQL systems. In: Proceedings of the VLDB Endowment. Conference paper. Association for Computing Machinery. März 2024. S. 685–698
Zhang, Yi, Jan Milan Deriu, George Katsogiannis-Meimarakis, Catherine Kosten, Georgia Koutrika, and Kurt Stockinger. 2024. “ScienceBenchmark : A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systems.” Conference paper. In Proceedings of the VLDB Endowment, 17:685–98. Association for Computing Machinery. https://doi.org/10.14778/3636218.3636225.
Zhang, Yi, et al. “ScienceBenchmark : A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systems.” Proceedings of the VLDB Endowment, vol. 17, no. 4, Association for Computing Machinery, 2024, pp. 685–98, https://doi.org/10.14778/3636218.3636225.
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.