Inter-rater agreement in the assessment of response to motor and cognitive rehabilitation of children and adolescents with epilepsy

Abstract

Introduction: To assess inter-rater agreement among child neurologists and psychiatrists on evaluation of response to physical and cognitive rehabilitation of children and adolescents with epilepsy.

Materials and methods: Five child neurologists/psychiatrists ("raters") were invited to draw 2-3 short case reports among those most commonly seen. 14 case histories were presented and raters used a structured questionnaire to report changes after selected rehabilitation programs. Response was coded as "Yes", "No", or "Uncertain" in different functional domains (Motor, Social, Alimentary, Communication, Personal Autonomy). Inter-rater agreement was measured using the kappa statistic. Raters where then asked to discuss any reason for disagreement. The test was repeated with different cases (16 case histories) adding a sixth rater, who had participated to the discussion.

Results: Even with this small number of cases, the agreement mostly ranged from poor to good in the first test (worse for Social, Personal Autonomy and Communication). Training improved agreement in almost all domains. There were no frank outliers. The agreement was lower with a specific approach (i.e. grouping "Uncertain" to "No") than with sensitive approach (i.e. grouping "Uncertain" to "Yes").

Discussion: The interpretation of patients' response to physical and cognitive rehabilitation tends to vary among Italian child neurologists/psychiatrists depending on measures and training procedures. Discussion and training improves agreement, although this is only a pilot study conducted using a non standardized questionnaire.