How do you design a reproducibility study?

We recommend conducting the reproducibility study at a minimum of 3 of the same sites that were included in the comparison study and are representative of the intended use of the waived test. To facilitate statistical analysis, the same number of untrained operators (likely 2 or 3) should be included at each site of the reproducibility study. For reproducibility study design and analysis, it is recommended that you follow FDA-recognized consensus standards (e.g. CLSI EP05, CLSI EP12). FDA recommends that you include the following sources of variability: different sites, different untrained operators, different days, different runs, different lots (if applicable), and a few replicates. If the candidate device is a unitized device, contact FDA through a Pre-Submission to discuss how you should evaluate repeatability.

Two possible study designs for evaluation of lot-to-lot variability are described below:

• Design 1: Include three different lots at each of three sites in the reproducibility study in such a way that the between-lot component can be evaluated.
• Design 2: Evaluate lot-to-lot variability in a separate small study at one internal site where patient (or surrogate) samples and controls are tested over a few days.

A reproducibility study design where each site uses a different lot is generally undesirable, especially for new technologies, because it would be impossible to determine whether observed differences are lot-related or site-related.

If specimens used with the candidate test are not stable (for example, capillary whole blood), attempts to use small-scale repeatability/reproducibility studies that use the intended use clinical samples should be explored (please contact FDA through a Pre-Submission to discuss study designs for precision/reproducibility studies).

We recommend that you include in the reproducibility study the following samples:

• For quantitative tests, the following levels of analyte should be included: close to the lower limit of the measuring interval, below the medical decision level (MDL), around the MDL, above the MDL, and close to the upper limit of the measuring interval. If the candidate device has more than one MDL, then samples with concentrations around these MDLs should be evaluated. It is understood that some tests will not have specific MDLs, but rather a range of values. In such cases, the reproducibility panel should contain samples scattered throughout the measuring interval of the candidate test.
• For binary qualitative tests with an analytical cutoff: true negative, close to the LoD, and moderate positive samples should be included.
• For binary qualitative tests with a clinical cutoff: true negative, high negative (close to C5), low positive (close to C95) and moderate positive samples should be included. C5 is a sample concentration which yields a positive result 5% of the time (and a negative result 95% of the time), and C95 is a sample concentration which yields a positive result 95% of the time (and a negative result 5% of the time).
• In addition, you should run the appropriate quality control samples associated with the candidate test.

We recommend discussing reproducibility study design as part of a Pre-Submission prior to conducting the reproducibility study.