Spot The Bot : a robust and efficient framework for the evaluation of conversational dialogue systems

Deriu, Jan Milan; Tuggener, Don; von Däniken, Pius; Campos, Jon Ander; Rodrigo, Alvaro; Belkacem, Thiziri; Soroa, Aitor; Agirre, Eneko; Cieliebak, Mark

doi:10.18653/v1/2020.emnlp-main.326

Please use this identifier to cite or link to this item: https://doi.org/10.21256/zhaw-21229

Publication type:	Conference paper
Type of review:	Peer review (publication)
Title:	Spot The Bot : a robust and efficient framework for the evaluation of conversational dialogue systems
Authors:	Deriu, Jan Milan Tuggener, Don von Däniken, Pius Campos, Jon Ander Rodrigo, Alvaro Belkacem, Thiziri Soroa, Aitor Agirre, Eneko Cieliebak, Mark
et. al:	No
DOI:	10.18653/v1/2020.emnlp-main.326 10.21256/zhaw-21229
Proceedings:	Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Page(s):	3971
Pages to:	3984
Conference details:	Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, 16-20 November 2020
Issue Date:	Nov-2020
Publisher / Ed. Institution:	Association for Computational Linguistics
Language:	English
Subjects:	Natural language processing; Chatbot; Dialogue system
Subject (DDC):	410.285: Computational linguistics
Abstract:	The lack of time efficient and reliable evaluation methods is hampering the development of conversational dialogue systems (chat bots). Evaluations that require humans to converse with chat bots are time and cost intensive, put high cognitive demands on the human judges, and tend to yield low quality results. In this work, we introduce Spot The Bot, a cost-efficient and robust evaluation framework that replaces human-bot conversations with conversations between bots. Human judges then only annotate for each entity in a conversation whether they think it is human or not (assuming there are humans participants in these conversations). These annotations then allow us to rank chat bots regarding their ability to mimic conversational behaviour of humans. Since we expect that all bots are eventually recognized as such, we incorporate a metric that measures which chat bot is able to uphold human-like behavior the longest, i.e.Survival Analysis. This metric has the ability to correlate a bot’s performance to certain of its characteristics (e.g.fluency or sensibleness), yielding interpretable results. The comparably low cost of our framework allows for frequent evaluations of chatbots during their evaluation cycle. We empirically validate our claims by applying Spot The Bot to three domains, evaluating several state-of-the-art chat bots, and drawing comparisonsto related work. The framework is released asa ready-to-use tool.
URI:	https://digitalcollection.zhaw.ch/handle/11475/21229
Fulltext version:	Published version
License (according to publishing contract):	CC BY 4.0: Attribution 4.0 International
Departement:	School of Engineering
Organisational Unit:	Institute of Computer Science (InIT)
Published as part of the ZHAW project:	LIHLITH - Learning to Interact with Humans by Lifelong Interaction with Humans
Appears in collections:	Publikationen School of Engineering

Files in This Item:

File	Description	Size	Format
2020_Deriu-etal_Spot-The-Bot_EMNLP.pdf		871.97 kB	Adobe PDF	View/Open

Show full item record

Deriu, J. M., Tuggener, D., von Däniken, P., Campos, J. A., Rodrigo, A., Belkacem, T., Soroa, A., Agirre, E., & Cieliebak, M. (2020). Spot The Bot : a robust and efficient framework for the evaluation of conversational dialogue systems [Conference paper]. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 3971–3984. https://doi.org/10.18653/v1/2020.emnlp-main.326

Deriu, J.M. et al. (2020) ‘Spot The Bot : a robust and efficient framework for the evaluation of conversational dialogue systems’, in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, pp. 3971–3984. Available at: https://doi.org/10.18653/v1/2020.emnlp-main.326.

J. M. Deriu et al., “Spot The Bot : a robust and efficient framework for the evaluation of conversational dialogue systems,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Nov. 2020, pp. 3971–3984. doi: 10.18653/v1/2020.emnlp-main.326.

DERIU, Jan Milan, Don TUGGENER, Pius VON DÄNIKEN, Jon Ander CAMPOS, Alvaro RODRIGO, Thiziri BELKACEM, Aitor SOROA, Eneko AGIRRE und Mark CIELIEBAK, 2020. Spot The Bot : a robust and efficient framework for the evaluation of conversational dialogue systems. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Conference paper. Association for Computational Linguistics. November 2020. S. 3971–3984

Deriu, Jan Milan, Don Tuggener, Pius von Däniken, Jon Ander Campos, Alvaro Rodrigo, Thiziri Belkacem, Aitor Soroa, Eneko Agirre, and Mark Cieliebak. 2020. “Spot the Bot : A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems.” Conference paper. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 3971–84. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.326.

Deriu, Jan Milan, et al. “Spot the Bot : A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems.” Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, 2020, pp. 3971–84, https://doi.org/10.18653/v1/2020.emnlp-main.326.