Psychiatry Research
Volume 166, Issue 2 , Pages 269-280 , 30 April 2009

Evaluating the reliability of multiple assessments of PTSD symptomatology: Multiple examiners, one patient

  • Domenic Cicchetti

      Affiliations

    • Yale University, Child Study Center and Departments of Psychiatry and Biometry, New Haven, CT 06512, United States
    • Corresponding Author InformationCorresponding author. Tel.: +1 203 488 6563; fax: +1 203 483 1123.
    • ,
  • Alan Fontana

      Affiliations

    • North East Program Evaluation Center (NEPEC), West Haven, CT 06516, United States
    • Yale University, Department of Psychiatry, New Haven, CT, United States
    • ,
  • Donald Showalter

      Affiliations

    • Yale University, Child Study Center and Departments of Psychiatry and Biometry, New Haven, CT 06512, United States

Received 14 February 2007 ,Revised 28 June 2007 ,Accepted 24 January 2008.

References 

  1. Allport GW. Personality: A Psychological Interpretation. New York: Holt; 1937;
  2. Baca-Garcia E, Blanco C, Saiz-Ruiz J, Diaz-Sastre C, Cicchetti DV. Assessment of reliability in the clinical evaluation among investigators in a multi-center clinical trial. Psychiatry Research. 2001;102:163–173
  3. Blake DD, Weathers FW, Nagy LM, Kaloupek DG, Klauminzer G, Charney DS, et al. A clinician rating scale for assessing current and lifetime PTSD: the Caps-1. Behavior Therapist. 1990;13:187–188
  4. Cicchetti DV. Assessing inter-rater reliability for rating scales: resolving some basic issues. British Journal of Psychiatry. 1976;129:452–456
  5. Cicchetti DV. The precision of reliability and validity estimates re-visited: distinguishing between clinical and statistical significance of sample size requirements. Journal of Clinical and Experimental Neuropsychology. 2001;23:695–700
  6. Cicchetti DV, Sparrow SS. Developing criteria for establishing interrater reliability of specific items: applications to assessment of adaptive behavior. American Journal of Mental Deficiency. 1981;86:127–137
  7. Cicchetti DV, Showalter D, McCarthy P. A computer program for calculating subject-by-subject kappa or weighted kappa coefficients. Educational and Psychological Measurement. 1990;50:153–158
  8. Cicchetti DV, Volkmar F, Sparrow SS, Cohen D, Fermanian J, Rourke BP. Assessing the reliability of clinical scales when the data have both nominal and ordinal features: proposed guidelines for neuropsychological assessments. Neuropsychology. 1992;14:673–686
  9. Cicchetti DV, Volkmar F, Klin A, Showalter D. Diagnosing autism using ICD-10 criteria: a comparison of neural networks and standard multivariate procedures. Child Neuropsychology. 1995;1:26–37
  10. Cicchetti DV, Showalter D, Rosenheck R. A new method for assessing interexaminer agreement when multiple ratings are made on a single subject: applications to the assessment of neuropsychiatric symptomatology. Psychiatry Research,. 1997;72:51–63
  11. Cicchetti DV, Rosenheck R, Showalter D, Charney D, Cramer J. Interrater reliability levels of multiple clinical examiners in the evaluation of a schizophrenic patient: quality of life; level of functioning; and neuropsychological symptomatology. The Clinical Neuropsychologist. 1999;13:157–170
  12. Cicchetti DV, Bronen R, Spencer S, Haut S, Berg A, Oliver P, et al. Rating scales, scales of measurement, issues of reliability: resolving some critical issues for clinicians and researchers. Journal of Nervous and Mental Disease. 2006;194:557–564
  13. Cicchetti, D.V., Lord, C., Koenig, K., Klin, A., Volkmar, F.R., in press. Reliability of the ADI-R: Multiple examiners evaluate a single case. Journal of Autism and Developmental Disorders.
  14. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. 1960;23:37–46
  15. Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin. 1968;70:213–220
  16. Demitrack MA, Faries D, Herrera JM, Debrota D, Potter WZ. The problem of measurement error in multiple clinical trials.. Psychopharmacology Bulletin. 1998;34:19–24
  17. Fleiss JL. Statistical Methods for Rates and Proportions. 2nd ed.. New York: Wiley; 1981;
  18. Fleiss JL, Cohen J, Everitt BS. Large sample standard errors of kappa and weighted kappa. Psychological Bulletin. 1969;72:323–327
  19. Fleiss JL, Nee JCM, Landis JR. Large sample variance of kappa in the case of different sets of raters.. Psychological Bulletin. 1979;86:974–977
  20. Fleiss JL, Levin B, Paik MC. Statistical Methods for Rates and Proportions. 3rd ed.. New York: Wiley; 2003;
  21. Grice JW, Jackson BJ, McDaniel BL. Bridging the idiographic–nomothetic divide: a follow-up study. Journal of Personality. 2006;74:1191–1218
  22. Grove WM, Andreasen NC, McDonald-Scott P, Keller MB, Shapiro RW. Reliability studies of psychiatric diagnosis: theory and practice. Archives of General Psychiatry. 1981;38:408–413
  23. Holschuh N. Randomization and design: I.. In:  Fienberg S,  Hinkley DV editor. R.A. Fisher: An appreciation. New York, NY: Springer-Verlag; 1980;p. 36–39
  24. Kay SR, Opler LA, Lindenmayer JP. Reliability and validity of the Positive and Negative Syndrome Scale for schizophrenics. Psychiatry Research. 1988;23:99–110
  25. Kraemer HC. Evaluating Medical Tests: Objective and Quantitative Guidelines. Newbury Park, California: Sage Publications; 1992;
  26. Landis JR, Koch GG. The measurement of agreement for categorical data. Biometrics. 1977;33:159–174
  27. Lindstrom E, Wieselgren IM, von Knorring L. Interrater reliability of the Structured Clinical Interview for the Positive and Negative Syndrome Scale for schizophrenia. Acta Psychiatrica Scandinavica. 1994;89:192–195
  28. McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psychological Methods. 1996;1:30–46
  29. Nathan PE, Andberg MM, Behan PO, Patch VD. Thirty two observers and one patient: a study of diagnostic reliability. Journal of Clinical Psychology. 1969;25:9–15
  30. Parker, R.M., 2002. Parker's Wine Buyer's Guide. Simon & Schuster, New York (6th ed.).
  31. Rockwood K, Strang D, MacKnight C, Downer R, Morris JC. Interrater reliability of the Clinical Dementia Rating in a multicenter trial.. Journal of the American geriatric Society. 2000;48:558–559
  32. Salsberg D. In: The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century. New York: Freeman; 2001;p. 1–8
  33. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin. 1979;86:420–428
  34. Szalai P. The statistics of agreement on a single rating category for a single item or object rated by multiple raters. Perceptual and Motor Skills. 1993;77:377–378
  35. Volkmar FR, Cicchetti DV, Dykens E, Sparrow SS, Leckman JF, Cohen DJ. An evaluation of the Autism Behavior Checklist. Journal of Autism and Developmental Disorders. 1988;18:81–97
  36. Weathers FW, Litz BT. Psychometric properties of the clinician-administered PTSD Scale, CAPS-1. PTSD Research Quarterly. 1994;5:2–6
  37. Wilson BA. Single-case experimental designs in rehabilitation. Journal of Clinical and Experimental Neuropsychology. 1987;9:527–544
  38. Windelbrandt W. Geschicte und Naturwissenschaft (History and science). In: Rektoratsreden der Universitat Strassbourgh (Presidential speeches at the University of Strassbourg). Strassbourgh: Heitz und Mundel; 1894;p. 193–208

PII: S0165-1781(08)00033-4

doi: 10.1016/j.psychres.2008.01.014

Psychiatry Research
Volume 166, Issue 2 , Pages 269-280 , 30 April 2009

Article Tools