(Dis)agreement among professionals on child protection assessments in cases of neglect : results from a multifactorial experimental vignette study in Switzerland

Gautschi, Joel

Publication type:	Conference other
Type of review:	Peer review (abstract)
Title:	(Dis)agreement among professionals on child protection assessments in cases of neglect : results from a multifactorial experimental vignette study in Switzerland
Authors:	Gautschi, Joel
et. al:	No
Conference details:	26rd Annual Conference of the Society for Social Work and Research (SSWR), Washington DC; USA (online), 12-16 January 2022
Issue Date:	14-Jan-2022
Language:	English
Subjects:	Child protection; Decision-making; Variability; Noise; Vignette study
Subject (DDC):	362.7: Youth services
Abstract:	Background/Purpose: Professionals in child protection are required to make decisions under uncertainty and are confronted with dilemmatic demands. Street-level bureaucrats (Lipsky 2010/1980), like child protection professionals, are assumed to have some discretion in their judgement and decision-making. While there are some studies on agreement between professionals in child protection assessments (cf. Keddel 2014), studies reporting statistical measures of agreement between professionals considering multiple cases are still scarce. This paper presents findings on the agreement between professionals in child protection assessments in Switzerland and exploratory findings on what might explain (dis-)agreement between them. Methods: The study used a multifactorial, experimental vignette design (factorial survey approach, cf. Rossi & Anderson, 1982). The vignettes described situations of suspected neglect of a toddler of a single mother and consisted of seven experimentally varied case factors. Out of all possible vignettes, 54 vignettes were selected using a fractional factorial sampling procedure (Kuhfeld, 2010), optimized for D-efficiency (uncorrelated factors, balanced factor levels). The 54 vignettes were split into 18 decks of 3 vignettes each. Vignette decks were randomly assigned to participants and vignettes were presented in randomized order. For each vignette, respondents determined a) risk assessment (DV1, 7-point scale), b) the likelihood of recommending an out-of-home-placement (DV2, 6-point scale). Data were collected in an online survey of professionals, responsible for child protection assessments, in German-speaking Switzerland. 543 professionals (response rate: 63%) from 159 organizations participated, and rated 1625 vignettes. Each of the 54 vignettes were rated by 24 to 37 respondents. Two-way random effects models, accounting for case-level and respondent-level variance were used to calculate intraclass correlation coefficients ICC (2,1) as a measurement for absolute agreement among respondents (ICC = 1 indicates perfect agreement, ICC = 0 indicates no agreement). Results: The intraclass correlation coefficient for the respondents’ risk assessment (DV1) was ICC (2,1) = .32 (95% CI [.26, .40], F(53, 26394) = 476.30, p < .001). For the out-of-home placement recommendation the intraclass correlation coefficient was ICC (2,1) = .24 (95% CI [.19, .31], F(53, 26394) = 306.30, p < .001). However, agreement between respondents varied by vignettes. Exploratory analyses showed that the lowest standard deviations for DV1 and DV2 occurred in vignettes with very consistent case factor levels. However, the consistency of case factor levels does not fully explain respondents’ (dis)agreement. Moreover, more severe case factor levels tended to be related to higher standard deviations for risk assessment (DV1) and to lower standard deviations for out-of-home placement recommendations (DV2). Conclusions/Implications: Compared to standards for clinical testing the ICC indicated poor agreement between professionals (ICC < .40, Cicchetti, 1994; ICC < .50, Portney, 2020). However, these standards should only be considered as rules of thumb and task-specific standards are required. ICC should be interpreted cautiously because they are affected by various statistical properties. Reasons for (dis-)agreement between professionals should be further investigated. Especially in ambiguous cases, the perspective of multiple professionals should be taken into account in child protection assessments and divergent opinions should be made transparent to decision-makers.
URI:	https://sswr.confex.com/sswr/2022/webprogram/Paper46544.html https://digitalcollection.zhaw.ch/handle/11475/27282
Fulltext version:	Published version
License (according to publishing contract):	Licence according to publishing contract
Departement:	Social Work
Organisational Unit:	Institute of Childhood, Youth and Family (IKJF)
Appears in collections:	Publikationen Soziale Arbeit

Files in This Item:

There are no files associated with this item.

Show full item record

Gautschi, J. (2022, January 14). (Dis)agreement among professionals on child protection assessments in cases of neglect : results from a multifactorial experimental vignette study in Switzerland. 26rd Annual Conference of the Society for Social Work and Research (SSWR), Washington DC; USA (Online), 12-16 January 2022. https://sswr.confex.com/sswr/2022/webprogram/Paper46544.html

Gautschi, J. (2022) ‘(Dis)agreement among professionals on child protection assessments in cases of neglect : results from a multifactorial experimental vignette study in Switzerland’, in 26rd Annual Conference of the Society for Social Work and Research (SSWR), Washington DC; USA (online), 12-16 January 2022. Available at: https://sswr.confex.com/sswr/2022/webprogram/Paper46544.html.

J. Gautschi, “(Dis)agreement among professionals on child protection assessments in cases of neglect : results from a multifactorial experimental vignette study in Switzerland,” in 26rd Annual Conference of the Society for Social Work and Research (SSWR), Washington DC; USA (online), 12-16 January 2022, Jan. 2022. [Online]. Available: https://sswr.confex.com/sswr/2022/webprogram/Paper46544.html

GAUTSCHI, Joel, 2022. (Dis)agreement among professionals on child protection assessments in cases of neglect : results from a multifactorial experimental vignette study in Switzerland. In: 26rd Annual Conference of the Society for Social Work and Research (SSWR), Washington DC; USA (online), 12-16 January 2022 [online]. Conference presentation. 14 Januar 2022. Verfügbar unter: https://sswr.confex.com/sswr/2022/webprogram/Paper46544.html

Gautschi, Joel. 2022. “(Dis)agreement among Professionals on Child Protection Assessments in Cases of Neglect : Results from a Multifactorial Experimental Vignette Study in Switzerland.” Conference presentation. In 26rd Annual Conference of the Society for Social Work and Research (SSWR), Washington DC; USA (Online), 12-16 January 2022. https://sswr.confex.com/sswr/2022/webprogram/Paper46544.html.

Gautschi, Joel. “(Dis)agreement among Professionals on Child Protection Assessments in Cases of Neglect : Results from a Multifactorial Experimental Vignette Study in Switzerland.” 26rd Annual Conference of the Society for Social Work and Research (SSWR), Washington DC; USA (Online), 12-16 January 2022, 2022, https://sswr.confex.com/sswr/2022/webprogram/Paper46544.html.