Childhood Adenotonsillectomy Trial

6.10.4 External Review of Scoring Reliability

The Data Coordinating Center and/or designated Polysomnography (PSG) Committee will establish methods for tracking scoring reliability and drift. This includes re-processing of records for reliability exercises and generation of summary data for each scorer. The overall mean for: OAI and AHI, sleep stage percentage, and arousal indexes will be calculated for each scorer over discrete time periods (monthly to quarterly) and reported to the PSG Subcommittee/Coordinating Center (DCC). Intra- and inter-reader differences in the summary data for RDI, sleep stages, and arousal indices will be determined. The same mean values will be calculated for each site in the same periods of the time. The CC will monitor intra- and inter-reader reliability to determine the threshold for requiring remediation, including retraining or removing a reader.

The PSG Committee, in conjunction with the Coordinating Center (DCC) may design a formal reliability study of scoring. For example, a sample of previously read PSG studies will be assigned a new “dummy” identification number and be assigned to the scorers for repeat scoring. Scoring of these studies will be integrated into the normal work flow to minimize their likelihood of being identified as a special study. Studies will be assigned to a different reader to define inter-reader reliability. Studies will be also assigned to the same reader at defined time periods to define intra-reader reliability.