1. ZHAW digitalcollection
  2. School of Engineering
  3. Publikationen
  4. Publikationen School of Engineering
Please use this identifier to cite or link to this item: https://doi.org/10.21256/zhaw-21229
Full metadata record
DC FieldValueLanguage
dc.contributor.authorDeriu, Jan Milan-
dc.contributor.authorTuggener, Don-
dc.contributor.authorvon Däniken, Pius-
dc.contributor.authorCampos, Jon Ander-
dc.contributor.authorRodrigo, Alvaro-
dc.contributor.authorBelkacem, Thiziri-
dc.contributor.authorSoroa, Aitor-
dc.contributor.authorAgirre, Eneko-
dc.contributor.authorCieliebak, Mark-
dc.date.accessioned2021-01-11T09:18:43Z-
dc.date.available2021-01-11T09:18:43Z-
dc.date.issued2020-11-
dc.identifier.urihttps://digitalcollection.zhaw.ch/handle/11475/21229-
dc.description.abstractThe lack of time efficient and reliable evaluation methods is hampering the development of conversational dialogue systems (chat bots). Evaluations that require humans to converse with chat bots are time and cost intensive, put high cognitive demands on the human judges, and tend to yield low quality results. In this work, we introduce Spot The Bot, a cost-efficient and robust evaluation framework that replaces human-bot conversations with conversations between bots. Human judges then only annotate for each entity in a conversation whether they think it is human or not (assuming there are humans participants in these conversations). These annotations then allow us to rank chat bots regarding their ability to mimic conversational behaviour of humans. Since we expect that all bots are eventually recognized as such, we incorporate a metric that measures which chat bot is able to uphold human-like behavior the longest, i.e.Survival Analysis. This metric has the ability to correlate a bot’s performance to certain of its characteristics (e.g.fluency or sensibleness), yielding interpretable results. The comparably low cost of our framework allows for frequent evaluations of chatbots during their evaluation cycle. We empirically validate our claims by applying Spot The Bot to three domains, evaluating several state-of-the-art chat bots, and drawing comparisonsto related work. The framework is released asa ready-to-use tool.de_CH
dc.language.isoende_CH
dc.publisherAssociation for Computational Linguisticsde_CH
dc.rightshttp://creativecommons.org/licenses/by/4.0/de_CH
dc.subjectNatural language processingde_CH
dc.subjectChatbotde_CH
dc.subjectDialogue systemde_CH
dc.subject.ddc410.285: Computerlinguistikde_CH
dc.titleSpot The Bot : a robust and efficient framework for the evaluation of conversational dialogue systemsde_CH
dc.typeKonferenz: Paperde_CH
dcterms.typeTextde_CH
zhaw.departementSchool of Engineeringde_CH
zhaw.organisationalunitInstitut für Informatik (InIT)de_CH
dc.identifier.doi10.18653/v1/2020.emnlp-main.326de_CH
dc.identifier.doi10.21256/zhaw-21229-
zhaw.conference.detailsConference on Empirical Methods in Natural Language Processing (EMNLP), Online, 16-20 November 2020de_CH
zhaw.funding.euNode_CH
zhaw.originated.zhawYesde_CH
zhaw.pages.end3984de_CH
zhaw.pages.start3971de_CH
zhaw.publication.statuspublishedVersionde_CH
zhaw.publication.reviewPeer review (Publikation)de_CH
zhaw.title.proceedingsProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)de_CH
zhaw.funding.snf174237de_CH
zhaw.webfeedSoftware Systemsde_CH
zhaw.webfeedNatural Language Processingde_CH
zhaw.funding.zhawLIHLITH - Learning to Interact with Humans by Lifelong Interaction with Humansde_CH
zhaw.author.additionalNode_CH
zhaw.display.portraitYesde_CH
Appears in collections:Publikationen School of Engineering

Files in This Item:
File Description SizeFormat 
2020_Deriu-etal_Spot-The-Bot_EMNLP.pdf871.97 kBAdobe PDFThumbnail
View/Open
Show simple item record
Deriu, J. M., Tuggener, D., von Däniken, P., Campos, J. A., Rodrigo, A., Belkacem, T., Soroa, A., Agirre, E., & Cieliebak, M. (2020). Spot The Bot : a robust and efficient framework for the evaluation of conversational dialogue systems [Conference paper]. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 3971–3984. https://doi.org/10.18653/v1/2020.emnlp-main.326
Deriu, J.M. et al. (2020) ‘Spot The Bot : a robust and efficient framework for the evaluation of conversational dialogue systems’, in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, pp. 3971–3984. Available at: https://doi.org/10.18653/v1/2020.emnlp-main.326.
J. M. Deriu et al., “Spot The Bot : a robust and efficient framework for the evaluation of conversational dialogue systems,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Nov. 2020, pp. 3971–3984. doi: 10.18653/v1/2020.emnlp-main.326.
DERIU, Jan Milan, Don TUGGENER, Pius VON DÄNIKEN, Jon Ander CAMPOS, Alvaro RODRIGO, Thiziri BELKACEM, Aitor SOROA, Eneko AGIRRE und Mark CIELIEBAK, 2020. Spot The Bot : a robust and efficient framework for the evaluation of conversational dialogue systems. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Conference paper. Association for Computational Linguistics. November 2020. S. 3971–3984
Deriu, Jan Milan, Don Tuggener, Pius von Däniken, Jon Ander Campos, Alvaro Rodrigo, Thiziri Belkacem, Aitor Soroa, Eneko Agirre, and Mark Cieliebak. 2020. “Spot the Bot : A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems.” Conference paper. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 3971–84. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.326.
Deriu, Jan Milan, et al. “Spot the Bot : A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems.” Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, 2020, pp. 3971–84, https://doi.org/10.18653/v1/2020.emnlp-main.326.


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.