1. ZHAW digitalcollection
  2. School of Engineering
  3. Publikationen
  4. Publikationen School of Engineering
Please use this identifier to cite or link to this item: https://doi.org/10.21256/zhaw-29262
Full metadata record
DC FieldValueLanguage
dc.contributor.authorHürlimann, Manuela-
dc.contributor.authorCieliebak, Mark-
dc.date.accessioned2023-12-01T16:14:42Z-
dc.date.available2023-12-01T16:14:42Z-
dc.date.issued2023-
dc.identifier.isbn978-954-452-088-5de_CH
dc.identifier.urihttps://aclanthology.org/2023.humeval-1.12de_CH
dc.identifier.urihttps://digitalcollection.zhaw.ch/handle/11475/29262-
dc.description.abstractThis paper describes the reproduction of a human evaluation in Language-Agnostic Meta- Learning for Low-Resource Text-to-Speech with Articulatory Features reported in Lux and Vu (2022). It is a contribution to the ReproNLP 2023 Shared Task on Reproducibility of Evaluations in NLP. The original evaluation assessed the naturalness of audio generated by different Text-to-Speech (TTS) systems for German, and our goal was to repeat the experiment with a different set of evaluators. We reproduced the evaluation based on data and instructions provided by the original authors, with some uncertainty concerning the randomisation of question order. Evaluators were recruited via email to relevant mailing lists and we received 157 responses over the course of three weeks. Our initial results show low reproducibility, but when we assume that the systems of the original and repeat evaluation experiment have been transposed, the reproducibility assessment improves markedly. We do not know if and at what point such a transposition happened; however, an initial analysis of our audio and video files provides some evidence that the system assignment in our repeat experiment is correct.de_CH
dc.language.isoende_CH
dc.publisherINCOMAde_CH
dc.rightshttp://creativecommons.org/licenses/by/4.0/de_CH
dc.subjectHuman evaluationde_CH
dc.subjectReproducibilityde_CH
dc.subject.ddc410.285: Computerlinguistikde_CH
dc.titleReproducing a comparative evaluation of German text-to-speech systemsde_CH
dc.typeKonferenz: Paperde_CH
dcterms.typeTextde_CH
zhaw.departementSchool of Engineeringde_CH
zhaw.organisationalunitCentre for Artificial Intelligence (CAI)de_CH
zhaw.publisher.placeShoumende_CH
dc.identifier.doi10.21256/zhaw-29262-
zhaw.conference.details14th International Conference Recent Advances in Natural Language Processing (RANLP), Varna, Bulgaria, 4-8 September 2023de_CH
zhaw.funding.euNode_CH
zhaw.originated.zhawYesde_CH
zhaw.pages.end144de_CH
zhaw.pages.start136de_CH
zhaw.parentwork.editorBelz, Anya-
zhaw.parentwork.editorPopovic, Maja-
zhaw.parentwork.editorReiter, Ehud-
zhaw.parentwork.editorThomson, Craig-
zhaw.parentwork.editorSedoc, João-
zhaw.publication.statuspublishedVersionde_CH
zhaw.publication.reviewEditorial reviewde_CH
zhaw.title.proceedingsProceedings of the 3rd Workshop on Human Evaluation of NLP Systemsde_CH
zhaw.webfeedDatalabde_CH
zhaw.webfeedNatural Language Processingde_CH
zhaw.author.additionalNode_CH
zhaw.display.portraitYesde_CH
Appears in collections:Publikationen School of Engineering

Files in This Item:
File Description SizeFormat 
2023_Huerlimann-Cieliebak_Comparative-evaluation-of-German-text-to-speech-systems.pdf469.58 kBAdobe PDFThumbnail
View/Open
Show simple item record
Hürlimann, M., & Cieliebak, M. (2023). Reproducing a comparative evaluation of German text-to-speech systems [Conference paper]. In A. Belz, M. Popovic, E. Reiter, C. Thomson, & J. Sedoc (Eds.), Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems (pp. 136–144). INCOMA. https://doi.org/10.21256/zhaw-29262
Hürlimann, M. and Cieliebak, M. (2023) ‘Reproducing a comparative evaluation of German text-to-speech systems’, in A. Belz et al. (eds) Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems. Shoumen: INCOMA, pp. 136–144. Available at: https://doi.org/10.21256/zhaw-29262.
M. Hürlimann and M. Cieliebak, “Reproducing a comparative evaluation of German text-to-speech systems,” in Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems, 2023, pp. 136–144. doi: 10.21256/zhaw-29262.
HÜRLIMANN, Manuela und Mark CIELIEBAK, 2023. Reproducing a comparative evaluation of German text-to-speech systems. In: Anya BELZ, Maja POPOVIC, Ehud REITER, Craig THOMSON und João SEDOC (Hrsg.), Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems [online]. Conference paper. Shoumen: INCOMA. 2023. S. 136–144. ISBN 978-954-452-088-5. Verfügbar unter: https://aclanthology.org/2023.humeval-1.12
Hürlimann, Manuela, and Mark Cieliebak. 2023. “Reproducing a Comparative Evaluation of German Text-to-Speech Systems.” Conference paper. In Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems, edited by Anya Belz, Maja Popovic, Ehud Reiter, Craig Thomson, and João Sedoc, 136–44. Shoumen: INCOMA. https://doi.org/10.21256/zhaw-29262.
Hürlimann, Manuela, and Mark Cieliebak. “Reproducing a Comparative Evaluation of German Text-to-Speech Systems.” Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems, edited by Anya Belz et al., INCOMA, 2023, pp. 136–44, https://doi.org/10.21256/zhaw-29262.


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.