| Publication type: | Conference other |
| Type of review: | Not specified |
| Title: | Automated assessment of writing and pragmatic competence : a critical review |
| Authors: | Gautschi, Curtis |
| et. al: | No |
| Proceedings: | 17th International Pragmatics Conference : abstracts |
| Page(s): | 261 |
| Conference details: | 17th International Pragmatics Conference (IPrA), Winterthur (online), 27 June - 2 July 2021 |
| Issue Date: | 2021 |
| Publisher / Ed. Institution: | International Pragmatics Association (IPrA) |
| Language: | English |
| Subjects: | Automated assessment; Pragmatics; Writing assessment |
| Subject (DDC): | 410.285: Computational linguistics 808: Rhetoric and writing |
| Abstract: | The automated scoring of essays and other written texts (AES), by means of natural language processing tools, is playing an ever-increasing role in the evaluation of writing ability. While able to provide quick, cost-effective and consistent feedback to learners of foreign languages (Shermis & Burstein, 2013), the use of computers to assess writing skills has been met with considerable scepticism (Condon, 2013). Of particular concern is the issue of construct relevance and representation in the automated assessment of writing (Weigle, 2013), which has relied heavily on a narrow range of linguistic features (e.g., McNamara, Crossley & McCarthy, 2010), cohesion markers or complexity indices (e.g., Coh-index; see Allen, Jacovina, & McNamara, 2016). Thus, while aspects of organizational competence (e.g., grammatical, lexical, syntax and text-level organization) are to some degree measurable, the valid measurement of pragmatic competence remains a challenge for AES (e.g., communicative facets including message conveyance, communicative intent, sensitivity to genre, and the socially constructed transfer of meaning - see, Allen et al., 2016; Bridgeman, Powers, Stone & Mollaun, 2011; Deane, 2013). This has been acknowledged by leaders in the high-stakes AES industry in their efforts to improve the measurement of pragmatic aspects in the communicative paradigm for both automated assessment of writing and speaking (e.g., Chen et al., 2018; Evanini, Hauck & Hakuta, 2017). This presentation seeks to problematize the issues raised above. First, an AES tool and CEFR-level prediction algorithm created and experimentally implemented at a major University of Applied Sciences in Switzerland as part of an online English placement test for first-year engineering students (Gautschi, 2020) is critically examined with respect to its design, results and coverage of pragmatic competence aspects. Second the AES tool is placed within the context of current AES research and approaches to modelling pragmatic competence, with a view to critically reviewing: • current scoring model approaches • claims of communicative and pragmatic competence assessment • advancements in scoring accuracy through deep learning / neural network techniques • the reliance on human-machine agreement for validation support. Finally, it is argued that the value of future AES in testing crucially requires the active involvement of linguists and applied linguists to safeguard construct coverage of pragmatic competence in high-stakes assessment of communicative language ability by means of automated writing assessment. |
| URI: | https://digitalcollection.zhaw.ch/handle/11475/23455 |
| Fulltext version: | Published version |
| License (according to publishing contract): | Licence according to publishing contract |
| Departement: | Applied Linguistics |
| Organisational Unit: | Institute of Language Competence (ILC) |
| Appears in collections: | Publikationen Angewandte Linguistik |
Files in This Item:
There are no files associated with this item.
Show full item record
Gautschi, C. (2021). Automated assessment of writing and pragmatic competence : a critical review [Conference presentation]. 17th International Pragmatics Conference : Abstracts, 261.
Gautschi, C. (2021) ‘Automated assessment of writing and pragmatic competence : a critical review’, in 17th International Pragmatics Conference : abstracts. International Pragmatics Association (IPrA), p. 261.
C. Gautschi, “Automated assessment of writing and pragmatic competence : a critical review,” in 17th International Pragmatics Conference : abstracts, 2021, p. 261.
GAUTSCHI, Curtis, 2021. Automated assessment of writing and pragmatic competence : a critical review. In: 17th International Pragmatics Conference : abstracts. Conference presentation. International Pragmatics Association (IPrA). 2021. S. 261
Gautschi, Curtis. 2021. “Automated Assessment of Writing and Pragmatic Competence : A Critical Review.” Conference presentation. In 17th International Pragmatics Conference : Abstracts, 261. International Pragmatics Association (IPrA).
Gautschi, Curtis. “Automated Assessment of Writing and Pragmatic Competence : A Critical Review.” 17th International Pragmatics Conference : Abstracts, International Pragmatics Association (IPrA), 2021, p. 261.
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.