1. ZHAW digitalcollection
  2. School of Engineering
  3. Publikationen
  4. Publikationen School of Engineering
Please use this identifier to cite or link to this item: https://doi.org/10.21256/zhaw-26130
Full metadata record
DC FieldValueLanguage
dc.contributor.authorDeriu, Jan Milan-
dc.contributor.authorTuggener, Don-
dc.contributor.authorvon Däniken, Pius-
dc.contributor.authorCieliebak, Mark-
dc.date.accessioned2022-11-18T14:26:33Z-
dc.date.available2022-11-18T14:26:33Z-
dc.date.issued2022-05-
dc.identifier.urihttps://digitalcollection.zhaw.ch/handle/11475/26130-
dc.description.abstractThis paper introduces an adversarial method to stress-test trained metrics for the evaluation of conversational dialogue systems. The method leverages Reinforcement Learning to find response strategies that elicit optimal scores from the trained metrics. We apply our method to test recently proposed trained metrics. We find that they all are susceptible to giving high scores to responses generated by rather simple and obviously flawed strategies that our method converges on. For instance, simply copying parts of the conversation context to form a response yields competitive scores or even outperforms responses written by humans.de_CH
dc.language.isoende_CH
dc.publisherAssociation for Computational Linguisticsde_CH
dc.rightshttp://creativecommons.org/licenses/by/4.0/de_CH
dc.subjectEvaluationde_CH
dc.subjectDiaglogue systemde_CH
dc.subject.ddc410.285: Computerlinguistikde_CH
dc.titleProbing the robustness of trained metrics for conversational dialogue systemsde_CH
dc.typeKonferenz: Paperde_CH
dcterms.typeTextde_CH
zhaw.departementSchool of Engineeringde_CH
zhaw.organisationalunitCentre for Artificial Intelligence (CAI)de_CH
dc.identifier.doi10.18653/v1/2022.acl-short.85de_CH
dc.identifier.doi10.21256/zhaw-26130-
zhaw.conference.details60th Annual Meeting of the Association for Computational Linguistics (ACL 2022), Dublin, Ireland, 22-27 May 2022de_CH
zhaw.funding.euNode_CH
zhaw.originated.zhawYesde_CH
zhaw.pages.end761de_CH
zhaw.pages.start750de_CH
zhaw.publication.statuspublishedVersionde_CH
zhaw.volume2de_CH
zhaw.publication.reviewPeer review (Publikation)de_CH
zhaw.title.proceedingsProceedings of the 60th Annual Meeting of the Association for Computational Linguisticsde_CH
zhaw.webfeedNatural Language Processingde_CH
zhaw.author.additionalNode_CH
zhaw.display.portraitYesde_CH
Appears in collections:Publikationen School of Engineering

Files in This Item:
File Description SizeFormat 
2022_Deriu-etal_Robustness-probing-of-trained-metrics.pdf307.81 kBAdobe PDFThumbnail
View/Open
Show simple item record
Deriu, J. M., Tuggener, D., von Däniken, P., & Cieliebak, M. (2022). Probing the robustness of trained metrics for conversational dialogue systems [Conference paper]. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2, 750–761. https://doi.org/10.18653/v1/2022.acl-short.85
Deriu, J.M. et al. (2022) ‘Probing the robustness of trained metrics for conversational dialogue systems’, in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pp. 750–761. Available at: https://doi.org/10.18653/v1/2022.acl-short.85.
J. M. Deriu, D. Tuggener, P. von Däniken, and M. Cieliebak, “Probing the robustness of trained metrics for conversational dialogue systems,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, May 2022, vol. 2, pp. 750–761. doi: 10.18653/v1/2022.acl-short.85.
DERIU, Jan Milan, Don TUGGENER, Pius VON DÄNIKEN und Mark CIELIEBAK, 2022. Probing the robustness of trained metrics for conversational dialogue systems. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Conference paper. Association for Computational Linguistics. Mai 2022. S. 750–761
Deriu, Jan Milan, Don Tuggener, Pius von Däniken, and Mark Cieliebak. 2022. “Probing the Robustness of Trained Metrics for Conversational Dialogue Systems.” Conference paper. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2:750–61. Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.acl-short.85.
Deriu, Jan Milan, et al. “Probing the Robustness of Trained Metrics for Conversational Dialogue Systems.” Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, vol. 2, Association for Computational Linguistics, 2022, pp. 750–61, https://doi.org/10.18653/v1/2022.acl-short.85.


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.